Deploy a read-only code-fleet-watch k3s CronJob #668

Closed
opened 2026-07-28 19:47:45 +00:00 by coilyco-ops · 12 comments
Member

Outcome

Deploy one read-only code-fleet-watch CronJob that inventories the canonical Forgejo fleet, runs modular repository hygiene checks, and sends transition-aware Telegram findings without mutating repositories or pull requests.

Ownership

Infrastructure owns the workload, Forgejo inventory, k3s manifest, scheduling, deployment tuning, secret projection, persistent finding state, and Telegram delivery.

AOS remains the sole owner of pre-commit policy evaluation and the public encoded leak-guard. The workload depends on agentic-os#774 for machine-readable fleet compliance. The private SSM term sweep is a deployment backstop and does not duplicate the public AOS ruleset.

Checks

  • private-term-sweep reads a hex-encoded denylist from an SSM SecureString, validates and decodes terms only in memory, and fixed-string scans current default-branch content. Findings expose only stable rule IDs, repository, path, line, and count.
  • stale-branches uses Forgejo APIs, excludes default and protected branches, and considers last commit activity plus associated open pull requests.
  • stale-pull-requests uses Forgejo APIs and reports age since activity, draft state, and merge or review state.
  • aos-precommit-coverage reports missing config, invalid config, missing canonical hooks, and declared .agentic-os-ignore opt-outs through the AOS evaluator.
  • aos-precommit-rollout reports coverage, explicit campaign-target progress, and latest-release freshness as separate dimensions.

Runtime shape

  • The CronJob enumerates every owned active repository with pagination and a nonzero inventory guard.
  • The CronJob performs API-only checks without cloning, then creates at most one shallow default-branch checkout per repository for content-backed checks.
  • The CronJob clones through the expected internal Forgejo Service authority into a size-limited emptyDir and rejects unexpected clone hosts.
  • ExternalSecrets project the Forgejo read token, encoded denylist, and Telegram credentials as mounted files rather than argv or environment values.
  • A small reconstructible PVC stores finding fingerprints and notification timestamps for new, reminder, recovery, and fleet-summary transitions.
  • Any SSM, inventory, authentication, clone, evaluator, or notification-path failure marks the sweep incomplete. The workload never reports a partial run as clean.

CronJob guardrails

Follow the established forgejo-package-upload-cleanup precedent:

  • concurrencyPolicy: Forbid
  • bounded retries, runtime, and job history
  • explicit CPU, memory, and ephemeral-storage requests and limits
  • non-root execution, read-only root filesystem, and dropped capabilities
  • automountServiceAccountToken: false
  • immutable Forgejo OCI image reference from the owning source publication

Notification behavior

  • New findings alert immediately.
  • Unchanged findings remind on a bounded interval.
  • Resolved findings emit one recovery.
  • Incomplete sweeps alert separately from policy findings.
  • A compact periodic fleet summary reports check coverage and AOS rollout progress.
  • Telegram never receives forbidden terms, matching lines, credentials, clone URLs, or private infrastructure identifiers.

Non-goals

  • The CronJob does not delete branches, close or update pull requests, edit pre-commit configuration, commit, push, or dispatch agents.
  • Separate Ward verbs and repository workflows own every remediation or rollout mutation.
  • The first version scans current default branches, not Git history, abandoned refs, issues, release assets, or packages.

Acceptance

  • A dry fixture run proves every check and notification transition without live mutation.
  • One successful scheduled run accounts for every active repository and records a complete inventory count.
  • A forced repository failure produces an incomplete-sweep result rather than a clean result.
  • The private denylist remains hex encoded in SSM and plaintext never appears in tracked files, pod logs, Kubernetes events, or Telegram.
  • The AOS rollout report consumes the AOS-owned evaluator from agentic-os#774 and carries no copied hook inventory or policy parser.
  • The runbook documents thresholds, explicit rollout targets, opt-outs, state recovery, SSM inventory, deployment, rollback, and safe live observation.
  • docs/FEATURES.md, repository validation, and the canonical pre-commit suite pass.
  • #666 preserves the retired host-side pin convergence timer and requires an isolated successor.
  • #282 records the older one-release rollout problem that this observer makes continuously visible.
  • agentic-os#488 supplied the guarded PR lifecycle and read helpers. This workload uses only the read side.
## Outcome Deploy one read-only `code-fleet-watch` CronJob that inventories the canonical Forgejo fleet, runs modular repository hygiene checks, and sends transition-aware Telegram findings without mutating repositories or pull requests. ## Ownership Infrastructure owns the workload, Forgejo inventory, k3s manifest, scheduling, deployment tuning, secret projection, persistent finding state, and Telegram delivery. AOS remains the sole owner of pre-commit policy evaluation and the public encoded `leak-guard`. The workload depends on [agentic-os#774](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/774) for machine-readable fleet compliance. The private SSM term sweep is a deployment backstop and does not duplicate the public AOS ruleset. ## Checks * `private-term-sweep` reads a hex-encoded denylist from an SSM SecureString, validates and decodes terms only in memory, and fixed-string scans current default-branch content. Findings expose only stable rule IDs, repository, path, line, and count. * `stale-branches` uses Forgejo APIs, excludes default and protected branches, and considers last commit activity plus associated open pull requests. * `stale-pull-requests` uses Forgejo APIs and reports age since activity, draft state, and merge or review state. * `aos-precommit-coverage` reports missing config, invalid config, missing canonical hooks, and declared `.agentic-os-ignore` opt-outs through the AOS evaluator. * `aos-precommit-rollout` reports coverage, explicit campaign-target progress, and latest-release freshness as separate dimensions. ## Runtime shape * The CronJob enumerates every owned active repository with pagination and a nonzero inventory guard. * The CronJob performs API-only checks without cloning, then creates at most one shallow default-branch checkout per repository for content-backed checks. * The CronJob clones through the expected internal Forgejo Service authority into a size-limited `emptyDir` and rejects unexpected clone hosts. * ExternalSecrets project the Forgejo read token, encoded denylist, and Telegram credentials as mounted files rather than argv or environment values. * A small reconstructible PVC stores finding fingerprints and notification timestamps for new, reminder, recovery, and fleet-summary transitions. * Any SSM, inventory, authentication, clone, evaluator, or notification-path failure marks the sweep incomplete. The workload never reports a partial run as clean. ## CronJob guardrails Follow the established `forgejo-package-upload-cleanup` precedent: * `concurrencyPolicy: Forbid` * bounded retries, runtime, and job history * explicit CPU, memory, and ephemeral-storage requests and limits * non-root execution, read-only root filesystem, and dropped capabilities * `automountServiceAccountToken: false` * immutable Forgejo OCI image reference from the owning source publication ## Notification behavior * New findings alert immediately. * Unchanged findings remind on a bounded interval. * Resolved findings emit one recovery. * Incomplete sweeps alert separately from policy findings. * A compact periodic fleet summary reports check coverage and AOS rollout progress. * Telegram never receives forbidden terms, matching lines, credentials, clone URLs, or private infrastructure identifiers. ## Non-goals * The CronJob does not delete branches, close or update pull requests, edit pre-commit configuration, commit, push, or dispatch agents. * Separate Ward verbs and repository workflows own every remediation or rollout mutation. * The first version scans current default branches, not Git history, abandoned refs, issues, release assets, or packages. ## Acceptance * A dry fixture run proves every check and notification transition without live mutation. * One successful scheduled run accounts for every active repository and records a complete inventory count. * A forced repository failure produces an incomplete-sweep result rather than a clean result. * The private denylist remains hex encoded in SSM and plaintext never appears in tracked files, pod logs, Kubernetes events, or Telegram. * The AOS rollout report consumes the AOS-owned evaluator from agentic-os#774 and carries no copied hook inventory or policy parser. * The runbook documents thresholds, explicit rollout targets, opt-outs, state recovery, SSM inventory, deployment, rollback, and safe live observation. * `docs/FEATURES.md`, repository validation, and the canonical pre-commit suite pass. ## Related history * #666 preserves the retired host-side pin convergence timer and requires an isolated successor. * #282 records the older one-release rollout problem that this observer makes continuously visible. * [agentic-os#488](https://forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/issues/488) supplied the guarded PR lifecycle and read helpers. This workload uses only the read side.
Author
Member

Spec addition from Kai: the private denylist also protects personal details from public code. Each hex rule declares scope public or all. Public-scoped rules run only when Forgejo explicitly reports the repository as public. Missing or ambiguous visibility makes the sweep incomplete, never clean. Findings remain safe metadata only and never include the decoded personal detail or matching line.

Spec addition from Kai: the private denylist also protects personal details from public code. Each hex rule declares scope `public` or `all`. Public-scoped rules run only when Forgejo explicitly reports the repository as public. Missing or ambiguous visibility makes the sweep incomplete, never clean. Findings remain safe metadata only and never include the decoded personal detail or matching line.
Author
Member

Infrastructure implementation landed on canonical main at 053769d.

Completed:

  • scope: public rules keep encoded personal details out of Forgejo-public repositories. scope: all rules scan the full fleet. Ambiguous visibility fails closed.
  • The observer inventories owner endpoints, checks stale PR review and merge context, checks stale unprotected branches, shallow-clones one default branch at a time, bounds file scanning, and records safe transition state.
  • ExternalSecrets, the private image pull, the CronJob, guarded denylist provisioning, immutable render/apply commands, rollback, state recovery, runbooks, and FEATURES are tracked.
  • 14 focused tests pass. The complete pre-commit suite passes. Main CI lint and trufflehog pass.
  • The immutable image publication passed, including remote manifest inspection: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/actions/runs/1728

Live boundary still pending:

  • An operator provisions /coilysiren/code-fleet-watch/denylist, applies forgejo.coilysiren.me/coilyco-flight-deck/code-fleet-watch:053769d62a2e5edd6bb402fab9371b59216c9a84, triggers one manual Job, and returns a log ending in complete=true with matching inventory coverage.
  • AOS pre-commit evaluation remains disabled until agentic-os#774 supplies the machine-readable evaluator and rollout target contract.

The issue remains open with interactive because the engineer did not deploy or iterate against live k3s.

Infrastructure implementation landed on canonical main at `053769d`. Completed: * `scope: public` rules keep encoded personal details out of Forgejo-public repositories. `scope: all` rules scan the full fleet. Ambiguous visibility fails closed. * The observer inventories owner endpoints, checks stale PR review and merge context, checks stale unprotected branches, shallow-clones one default branch at a time, bounds file scanning, and records safe transition state. * ExternalSecrets, the private image pull, the CronJob, guarded denylist provisioning, immutable render/apply commands, rollback, state recovery, runbooks, and FEATURES are tracked. * 14 focused tests pass. The complete pre-commit suite passes. Main CI lint and trufflehog pass. * The immutable image publication passed, including remote manifest inspection: https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/actions/runs/1728 Live boundary still pending: * An operator provisions `/coilysiren/code-fleet-watch/denylist`, applies `forgejo.coilysiren.me/coilyco-flight-deck/code-fleet-watch:053769d62a2e5edd6bb402fab9371b59216c9a84`, triggers one manual Job, and returns a log ending in `complete=true` with matching inventory coverage. * AOS pre-commit evaluation remains disabled until agentic-os#774 supplies the machine-readable evaluator and rollout target contract. The issue remains open with `interactive` because the engineer did not deploy or iterate against live k3s.
Author
Member

Live deployment completed.

  • SSM denylist version 1 is provisioned with one public-scope encoded rule.
  • Manifest type correction landed at b6b2158. API credential correction landed at c5ce201. The complete pre-commit suite passed for both.
  • Both ExternalSecrets report SecretSynced and Ready. The 64 MiB state PVC is Bound.
  • Manual Job code-fleet-watch-manual-20260728204657 completed 1/1 in 24 seconds. Its terminal log is: code-fleet-watch repos=32 findings=11 errors=0 complete=true.
  • The CronJob is unsuspended on schedule 17 */6 * * * UTC and pins image 053769d62a.

The AOS evaluator remains disabled pending agentic-os#774, so this issue stays open for that acceptance item.

Live deployment completed. * SSM denylist version 1 is provisioned with one public-scope encoded rule. * Manifest type correction landed at b6b2158. API credential correction landed at c5ce201. The complete pre-commit suite passed for both. * Both ExternalSecrets report SecretSynced and Ready. The 64 MiB state PVC is Bound. * Manual Job code-fleet-watch-manual-20260728204657 completed 1/1 in 24 seconds. Its terminal log is: code-fleet-watch repos=32 findings=11 errors=0 complete=true. * The CronJob is unsuspended on schedule 17 */6 * * * UTC and pins image 053769d62a2e5edd6bb402fab9371b59216c9a84. The AOS evaluator remains disabled pending agentic-os#774, so this issue stays open for that acceptance item.
Author
Member

Cadence update landed at ceb69a0 and is live. code-fleet-watch now runs once daily at 00:17 UTC with schedule 17 0 * * *. The CronJob is unsuspended and retains the proven immutable image.

Cadence update landed at ceb69a0 and is live. code-fleet-watch now runs once daily at 00:17 UTC with schedule 17 0 * * *. The CronJob is unsuspended and retains the proven immutable image.
Author
Member

Compact denylist alerts are live. Observer commit 8085905 renders term findings as !alias@owner/repo@branch:/path#Lline and silently retires state for removed rule IDs. Operator commit 69d5fe7 adds tracked ExternalSecret refresh after SSM rotation. SSM denylist version 2 now uses the semantic alias job-mention. Image publication, lint, trufflehog, and the 15 focused tests passed. Manual Job code-fleet-watch-manual-20260728220519 completed in 21 seconds with repos=32 findings=11 errors=0 complete=true. The daily CronJob is resumed on the new immutable image.

Compact denylist alerts are live. Observer commit 8085905 renders term findings as !alias@owner/repo@branch:/path#Lline and silently retires state for removed rule IDs. Operator commit 69d5fe7 adds tracked ExternalSecret refresh after SSM rotation. SSM denylist version 2 now uses the semantic alias job-mention. Image publication, lint, trufflehog, and the 15 focused tests passed. Manual Job code-fleet-watch-manual-20260728220519 completed in 21 seconds with repos=32 findings=11 errors=0 complete=true. The daily CronJob is resumed on the new immutable image.
Author
Member

Readable compact term locators are live at e7f358b. Telegram term findings now render as ! alias | owner/repo | path, with default branch and line number omitted from both finding state and output. Image publication, lint, trufflehog, the local image build, and all 15 focused tests passed. Manual Job code-fleet-watch-manual-20260728221631 completed in 23 seconds with repos=32 findings=11 errors=0 complete=true. The daily CronJob is resumed on the new immutable image.

Readable compact term locators are live at e7f358b. Telegram term findings now render as ! alias | owner/repo | path, with default branch and line number omitted from both finding state and output. Image publication, lint, trufflehog, the local image build, and all 15 focused tests passed. Manual Job code-fleet-watch-manual-20260728221631 completed in 23 seconds with repos=32 findings=11 errors=0 complete=true. The daily CronJob is resumed on the new immutable image.
Author
Member

Final compact locator format is live at 571f108: !alias@owner/repo@path. The default branch and line remain omitted. Image publication and trufflehog passed. The superseded lint run was cancelled by a newer main push, whose replacement lint and trufflehog runs passed. Fleet smoke code-fleet-watch-manual-20260728222922 completed with repos=32 findings=11 errors=0 complete=true. A separate formatter-plus-Telegram smoke Job completed 1/1 and sent !job-mention@smoke/test@README.md.

Final compact locator format is live at 571f108: !alias@owner/repo@path. The default branch and line remain omitted. Image publication and trufflehog passed. The superseded lint run was cancelled by a newer main push, whose replacement lint and trufflehog runs passed. Fleet smoke code-fleet-watch-manual-20260728222922 completed with repos=32 findings=11 errors=0 complete=true. A separate formatter-plus-Telegram smoke Job completed 1/1 and sent !job-mention@smoke/test@README.md.
Author
Member

Product-policy support landed on canonical main at c28d209.

  • Rule IDs may use a class prefix such as product:job-mention. The prefix selects repositories carrying the matching Forgejo topic, policy-product, so deployment JSON carries no repository roster.
  • ignore_dot_paths: true skips root dot-prefixed paths such as .forgejo/ and .github/ for that rule while retaining ordinary source and documentation scanning.
  • The corrected product roster is cli-guard, ward, agent-proxy, ward-mcp, and lunch-money-k8s. Reference MCP repositories and agent-compose remain outside.
  • 20 focused tests, the full pre-commit suite, local image build, lint, trufflehog, and immutable image publication passed.

Live activation remains pending. All five topic writes returned 403 before mutation because AOSGuard uses the bot token for the admin-only topic route. agentic-os#495 now carries the evidence. SSM denylist overwrite and k3s image rollout also remain behind their required operator confirmations.

Product-policy support landed on canonical main at `c28d209`. * Rule IDs may use a class prefix such as `product:job-mention`. The prefix selects repositories carrying the matching Forgejo topic, `policy-product`, so deployment JSON carries no repository roster. * `ignore_dot_paths: true` skips root dot-prefixed paths such as `.forgejo/` and `.github/` for that rule while retaining ordinary source and documentation scanning. * The corrected product roster is `cli-guard`, `ward`, `agent-proxy`, `ward-mcp`, and `lunch-money-k8s`. Reference MCP repositories and `agent-compose` remain outside. * 20 focused tests, the full pre-commit suite, local image build, lint, trufflehog, and immutable image publication passed. Live activation remains pending. All five topic writes returned 403 before mutation because AOSGuard uses the bot token for the admin-only topic route. agentic-os#495 now carries the evidence. SSM denylist overwrite and k3s image rollout also remain behind their required operator confirmations.
Author
Member

Landed the policy ownership correction in 7843853. Code Fleet Watch now reads repository classes only from metadata.policy_classes..repositories in the private SSM denylist document. Prefixed rules fail when their class is missing, malformed selectors and duplicates fail validation, the provisioner invokes the owning rules loader, and the sweep no longer reads Forgejo topics. Local pre-commit, the focused 20-test suite, and the container image build pass. The live SSM rotation and k3s rollout remain pending explicit operator confirmation because both overwrite live state.

Landed the policy ownership correction in 7843853. Code Fleet Watch now reads repository classes only from metadata.policy_classes.<class>.repositories in the private SSM denylist document. Prefixed rules fail when their class is missing, malformed selectors and duplicates fail validation, the provisioner invokes the owning rules loader, and the sweep no longer reads Forgejo topics. Local pre-commit, the focused 20-test suite, and the container image build pass. The live SSM rotation and k3s rollout remain pending explicit operator confirmation because both overwrite live state.
Author
Member

Landed and deployed the product-repository policy layer.

  • 5bda852 moves the public product roster into tracked deploy/code-fleet-watch/config.json and adds compact SSM document defaults.
  • ebfbd5d makes each matched path a distinct term finding while migrating the old state keys without false recovery alerts.
  • SSM denylist version 5 contains 28 base16-only rules. Public repository identities are no longer duplicated in SSM metadata.
  • The CronJob now runs immutable image ebfbd5d6cb5949633db1c8e4ba2b5b25c499ca3c on the existing daily schedule. Image, lint, and trufflehog workflows passed.
  • Manual job code-fleet-watch-manual-20260729023641 completed with 32/32 repositories swept, 38 raw occurrences, and 0 errors. Telegram enumerated 32 path-level findings after intentionally collapsing repeated lines in the same path: agent-proxy 3, cli-guard 5, lunch-money-k8s 8, ward 9, ward-mcp 7.
  • Ward test fixtures were neutralized in 70e60bb (ward#1614), and the subsequent Ward promote and release workflows passed.
Landed and deployed the product-repository policy layer. * `5bda852` moves the public product roster into tracked `deploy/code-fleet-watch/config.json` and adds compact SSM document defaults. * `ebfbd5d` makes each matched path a distinct term finding while migrating the old state keys without false recovery alerts. * SSM denylist version 5 contains 28 base16-only rules. Public repository identities are no longer duplicated in SSM metadata. * The CronJob now runs immutable image `ebfbd5d6cb5949633db1c8e4ba2b5b25c499ca3c` on the existing daily schedule. Image, lint, and trufflehog workflows passed. * Manual job `code-fleet-watch-manual-20260729023641` completed with 32/32 repositories swept, 38 raw occurrences, and 0 errors. Telegram enumerated 32 path-level findings after intentionally collapsing repeated lines in the same path: agent-proxy 3, cli-guard 5, lunch-money-k8s 8, ward 9, ward-mcp 7. * Ward test fixtures were neutralized in `70e60bb` (ward#1614), and the subsequent Ward promote and release workflows passed.
Author
Member

Rolling Forgejo issue reporting is landed and live.

  • 30f8df6 adds an observer init container and a separately credentialed reporter container. The observer mounts only the all-read token. The reporter mounts only the dedicated write:issue PAT. They exchange sanitized alias/path metadata through a pod-local volume.
  • Immutable image 30f8df6e81c9d756909d166e198ee066fc5e314d is deployed on the daily schedule. Image, lint, and trufflehog workflows all passed.
  • The PAT code-fleet-watch-issues is stored at /forgejo/coilyco-ops/code-fleet-watch-issue-token. Read access and create validation passed without creating a probe issue.
  • First sweep code-fleet-watch-manual-20260729032256 completed with 32 repositories, 38 findings, 0 errors, and created five rolling issues: agent-proxy#65, cli-guard#266, lunch-money-k8s#35, ward#1616, and ward-mcp#37.
  • Second sweep code-fleet-watch-manual-20260729032433 completed with the same coverage and reported unchanged for all five issues. Each issue remains open with zero comments, proving the no-op path created no duplicate issue or comment.
Rolling Forgejo issue reporting is landed and live. * `30f8df6` adds an observer init container and a separately credentialed reporter container. The observer mounts only the all-read token. The reporter mounts only the dedicated `write:issue` PAT. They exchange sanitized alias/path metadata through a pod-local volume. * Immutable image `30f8df6e81c9d756909d166e198ee066fc5e314d` is deployed on the daily schedule. Image, lint, and trufflehog workflows all passed. * The PAT `code-fleet-watch-issues` is stored at `/forgejo/coilyco-ops/code-fleet-watch-issue-token`. Read access and create validation passed without creating a probe issue. * First sweep `code-fleet-watch-manual-20260729032256` completed with 32 repositories, 38 findings, 0 errors, and created five rolling issues: [agent-proxy#65](https://forgejo.coilysiren.me/coilyco-flight-deck/agent-proxy/issues/65), [cli-guard#266](https://forgejo.coilysiren.me/coilyco-flight-deck/cli-guard/issues/266), [lunch-money-k8s#35](https://forgejo.coilysiren.me/coilyco-flight-deck/lunch-money-k8s/issues/35), [ward#1616](https://forgejo.coilysiren.me/coilyco-flight-deck/ward/issues/1616), and [ward-mcp#37](https://forgejo.coilysiren.me/coilyco-flight-deck/ward-mcp/issues/37). * Second sweep `code-fleet-watch-manual-20260729032433` completed with the same coverage and reported `unchanged` for all five issues. Each issue remains open with zero comments, proving the no-op path created no duplicate issue or comment.
Author
Member

Closing. The workload this issue deployed is retired, tracked at #830.

The observer ran daily from 2026-07-29 to 2026-08-14. The Telegram notification surface described here was never the shipped one, the reporter used rolling Forgejo issues instead. Those five managed issues are now closed with a retirement note.

Closing. The workload this issue deployed is retired, tracked at #830. The observer ran daily from 2026-07-29 to 2026-08-14. The Telegram notification surface described here was never the shipped one, the reporter used rolling Forgejo issues instead. Those five managed issues are now closed with a retirement note.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/infrastructure#668
No description provided.