coilyco-flight-deck/infrastructure
2
0
Fork 0
Code Issues 118 Pull requests Projects Releases Packages Wiki Activity Actions 2

Spec ser8 Forgejo Actions build/test runners without moving deploy runner #507

New issue
Open
opened 2026-07-09 21:34:19 +00:00 by coilyco-ops · 1 comment
coilyco-ops commented 2026-07-09 21:34:19 +00:00
Member
Copy link

Design-only request from Kai: spec moving/adding the normal Forgejo Actions build/test runners on ser8. Do not implement in this run.

Intent:

  • Use ser8 as the ops/CI plane and likely double build/test runner capacity.
  • Keep the deploy:host runner on kai-server for now because it relies on in-cluster ServiceAccount auth to deploy into kai-server k3s.
  • Avoid a cross-cluster deploy credential redesign in this spec. That is a later decision.

Current kai-server shape to compare against:

  • deploy/forgejo-runner.yml - StatefulSet in namespace forgejo, replicas: 4, per-pod DinD sidecar, labels are the stock runner labels (jobs use runs-on: docker).
  • Runner registration currently talks to http://forgejo.forgejo.svc.cluster.local, which only works inside kai-server's cluster.
  • DinD uses --insecure-registry=192.168.0.194:30500, the kai-server in-cluster registry.
  • deploy/forgejo-runner-deploy.yml - separate deploy:host runner with deployer ServiceAccount and in-cluster kubectl/helm. This is explicitly out of scope for the first ser8 move.

Questions the spec must answer:

  1. Registration path - How should ser8 runners reach Forgejo? Options likely include Forgejo's tailnet/public URL rather than forgejo.forgejo.svc.cluster.local. Decide the URL and TLS/auth implications.
  2. Labels and scheduling - Should ser8 runners share the existing docker label so capacity is automatic, or use a new label like docker-ser8 for a canary period? Recommend rollout sequence.
  3. Registry access - Jobs currently build/push to 192.168.0.194:30500. From ser8, decide whether to reach that registry over tailnet/LAN, mirror a registry on ser8, or publish elsewhere. Include failure modes and whether --insecure-registry remains acceptable.
  4. Secrets - ser8 does not currently have external-secrets/SSM in the same shape as kai-server. Decide how the runner registration token/API token is provided without leaking it into tracked files. Prefer a bounded token/Secret flow.
  5. Namespace and manifests - Recommend ser8 namespace name and manifest ownership. Likely an infra-owned deploy/forgejo-runner-ser8.yml or an ansible/fleet-reachability-style per-cluster rollout. Be explicit.
  6. Resource sizing - Current kai-server runners are 4 replicas, each runner 512Mi limit + DinD 2Gi limit. Recommend ser8 replica count and limits based on 57Gi RAM and 16 cores without starving o11y.
  7. Observability - Add labels/metrics/logging so runner capacity and failure rate are visible in ser8 o11y.
  8. Rollback - How to disable ser8 runners quickly without affecting kai-server deploy runner or existing build runners.

Deliverable:

  • A comment with the recommended architecture, rollout plan, and risks.
  • Follow-up implementation issue(s) only after the spec is posted.

Acceptance for the eventual implementation is not part of this issue, but the design should enable a first canary ser8 docker runner, then scaling to additional replicas once image push and checkout/build workloads are proven.

Design-only request from Kai: spec moving/adding the normal Forgejo Actions build/test runners on ser8. Do not implement in this run. Intent: * Use ser8 as the ops/CI plane and likely double build/test runner capacity. * Keep the `deploy:host` runner on kai-server for now because it relies on in-cluster ServiceAccount auth to deploy into kai-server k3s. * Avoid a cross-cluster deploy credential redesign in this spec. That is a later decision. Current kai-server shape to compare against: * `deploy/forgejo-runner.yml` - StatefulSet in namespace `forgejo`, `replicas: 4`, per-pod DinD sidecar, labels are the stock runner labels (jobs use `runs-on: docker`). * Runner registration currently talks to `http://forgejo.forgejo.svc.cluster.local`, which only works inside kai-server's cluster. * DinD uses `--insecure-registry=192.168.0.194:30500`, the kai-server in-cluster registry. * `deploy/forgejo-runner-deploy.yml` - separate `deploy:host` runner with `deployer` ServiceAccount and in-cluster kubectl/helm. This is explicitly out of scope for the first ser8 move. Questions the spec must answer: 1. **Registration path** - How should ser8 runners reach Forgejo? Options likely include Forgejo's tailnet/public URL rather than `forgejo.forgejo.svc.cluster.local`. Decide the URL and TLS/auth implications. 2. **Labels and scheduling** - Should ser8 runners share the existing `docker` label so capacity is automatic, or use a new label like `docker-ser8` for a canary period? Recommend rollout sequence. 3. **Registry access** - Jobs currently build/push to `192.168.0.194:30500`. From ser8, decide whether to reach that registry over tailnet/LAN, mirror a registry on ser8, or publish elsewhere. Include failure modes and whether `--insecure-registry` remains acceptable. 4. **Secrets** - ser8 does not currently have external-secrets/SSM in the same shape as kai-server. Decide how the runner registration token/API token is provided without leaking it into tracked files. Prefer a bounded token/Secret flow. 5. **Namespace and manifests** - Recommend ser8 namespace name and manifest ownership. Likely an infra-owned `deploy/forgejo-runner-ser8.yml` or an ansible/fleet-reachability-style per-cluster rollout. Be explicit. 6. **Resource sizing** - Current kai-server runners are 4 replicas, each runner 512Mi limit + DinD 2Gi limit. Recommend ser8 replica count and limits based on 57Gi RAM and 16 cores without starving o11y. 7. **Observability** - Add labels/metrics/logging so runner capacity and failure rate are visible in ser8 o11y. 8. **Rollback** - How to disable ser8 runners quickly without affecting kai-server deploy runner or existing build runners. Deliverable: * A comment with the recommended architecture, rollout plan, and risks. * Follow-up implementation issue(s) only after the spec is posted. Acceptance for the eventual implementation is not part of this issue, but the design should enable a first canary ser8 `docker` runner, then scaling to additional replicas once image push and checkout/build workloads are proven.
coilyco-ops commented 2026-07-09 21:38:13 +00:00
Author
Member
Copy link

🔎 ward agent advisor

ward agent advisor --harness codex ran a one-shot standard research pass on this question:

Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue.

Recommendation

The clean first design is split-plane, same Forgejo:

  • Keep deploy/forgejo-runner-deploy.yml and the deploy:host runner on kai-server unchanged.
  • Add a separate ser8 runner StatefulSet for normal runs-on: docker build/test capacity.
  • Treat ser8 as a second runner pool for the same Forgejo instance, not as a second deploy plane.
  • Keep the existing kai-server runners during the canary. Do not move or resize the deploy runner in this change.

This matches the repo's current shape:

  • deploy/forgejo-runner.yml is today a 4-replica DinD StatefulSet in forgejo, registering against http://forgejo.forgejo.svc.cluster.local and pushing to the in-cluster registry at 192.168.0.194:30500.
  • deploy/forgejo-runner-deploy.yml is already a separate deploy:host surface with a deployer ServiceAccount and is explicitly the wrong thing to move first.
  • docs/fleet-reachability.md confirms ser8 and kai-server are separate k3s clusters, and that ser8 does not have external-secrets today.

Decisions

1. Registration path

Use https://forgejo.coilysiren.me/ as FORGEJO_INSTANCE_URL for ser8 runners.

Why this is the right default:

  • The current forgejo.forgejo.svc.cluster.local address is cluster-local to kai-server and cannot work from ser8.
  • The public HTTPS URL is already Forgejo's stable external root URL in this repo's manifests and docs.
  • It avoids introducing a second reachability dependency just for runner registration.
  • It uses existing public TLS, so the ser8 runner image and DinD sidecar do not need a custom CA bundle just to talk to Forgejo.

TLS/auth implications:

  • Registration and runner polling happen over normal HTTPS to forgejo.coilysiren.me.
  • No mTLS is needed for this first step.
  • The secret on ser8 should be a runner-registration secret, not a general-purpose Forgejo admin/API token, if Forgejo's registration flow can support it for this pool.
  • If a broader token is still required for the current init-container pattern, keep it in a Kubernetes Secret created out-of-band from SSM or operator input, never in tracked YAML.

Why not the tailnet URL first:

  • It is viable as a fallback, but it adds DNS/routing assumptions between ser8 pods and the Tailscale path.
  • For the first move, public HTTPS is the lowest-complexity path and matches how users already reach Forgejo.

2. Labels and scheduling

Do not put ser8 straight onto the shared docker label. Start with a canary label, then promote.

Recommended rollout:

  1. First ser8 runner advertises docker-ser8 only.
  2. Run a small set of checkout/build/caching/image-push smoke workflows on docker-ser8.
  3. Once registry push/pull is proven, add the normal docker label to that same runner pool.
  4. Only then scale ser8 replicas up.

Reasoning:

  • If ser8 shares docker immediately, any existing image-build workflow can land there before registry reachability is solved.
  • The current docker pool is load-bearing. A bad first cut would turn random CI red instead of creating an isolated canary lane.

Promotion target:

  • After canary, ser8 runners should also carry docker so capacity becomes automatic.
  • The long-term steady state can be a mixed pool: kai-server docker runners + ser8 docker runners + kai-server-only deploy:host.

3. Registry access

Do not mirror a second registry on ser8, and do not switch package registries in this first move. Keep kai-server's registry as the source of truth, but give ser8 a remote-safe path to it.

Best first design:

  • Keep the existing registry on kai-server as the image source used by deploys.
  • Expose that registry to ser8 over a named remote path instead of the LAN-only 192.168.0.194:30500 literal.
  • Prefer a TLS endpoint for ser8's DinD daemon, not another insecure plain-HTTP hop if that can be avoided.

Why this over the alternatives:

  • Mirroring a registry on ser8 creates a new replication/cutover problem and makes deployment image provenance harder to reason about.
  • Publishing somewhere else such as Forgejo packages or GHCR is a broader cross-repo rollout. It changes app image refs and pull-secret handling across downstream repos, which is larger than this spec needs.
  • Keeping 192.168.0.194:30500 only works from kai-server/LAN-adjacent paths. It is the wrong address from ser8.

Failure modes to call out:

  • The registry remains physically hosted on kai-server, so ser8 builds still depend on kai-server reachability.
  • If the remote registry path is still plain HTTP plus --insecure-registry, that remains acceptable only as a temporary canary compromise. It is brittle and not a good permanent cross-site shape.
  • If the registry path is fronted with TLS, certificate trust and hostname stability get much better, but push latency becomes dependent on the WAN/tailnet hop.

Practical recommendation:

  • Canary phase: a remote path to the existing kai-server registry is enough.
  • Permanent phase: prefer a TLS name for the registry so ser8 does not need to normalize insecure-registry behavior forever.

4. Secrets

Because ser8 lacks external-secrets, the secret flow should be bounded and imperative, mirroring the existing ser8 pattern used elsewhere in this repo.

Recommended flow:

  • Create the ser8 runner secret as a normal Kubernetes Secret at apply time.
  • Materialize it from SSM or operator input on the control host, not from tracked YAML.
  • Store the minimum credential needed to register runners.

Preference order:

  1. Best: a runner-registration secret scoped to this runner pool.
  2. Second-best: a dedicated Forgejo token used only to mint registration tokens for this pool.
  3. Avoid: reusing a broad admin token when a narrower token works.

Operational note:

  • The repo already uses this style for ser8-specific cluster rollout in docs/fleet-reachability.md: resolve from SSM on the control host, then create the in-cluster secret imperatively.
  • The same pattern fits here.

5. Namespace and manifest ownership

Keep this infra-owned and manifest-owned in this repo.

Recommended ownership:

  • Add a dedicated manifest such as deploy/forgejo-runner-ser8.yml.
  • Keep the namespace name forgejo on ser8 as well.

Why:

  • The workload is still cluster infrastructure, not app-specific deploy config.
  • A separate file avoids accidental coupling to the kai-server-only assumptions in deploy/forgejo-runner.yml.
  • Reusing the forgejo namespace keeps operational queries symmetric across clusters.

Rollout method:

  • First implementation should be a human-run cluster apply, not something hidden inside ansible-sync.
  • If a later ansible wrapper is added, it should look like fleet-reachability: explicit, per-cluster, and confirm-before-apply.

6. Resource sizing

Current kai-server per-replica limits are effectively:

  • runner: 1 CPU / 512 MiB
  • DinD: 2 CPU / 2 GiB
  • total per replica: about 3 CPU / 2.5 GiB limit

For ser8 (57 GiB RAM, 16 cores), the safest recommendation is:

  • Canary: replicas: 1, same limits as kai-server.
  • First scale-up: replicas: 2 after checkout/build/image-push smoke passes.
  • Steady-state target: replicas: 3 on ser8 before considering 4.

Why 3 is the right initial target:

  • 3 replicas cap at about 9 CPU / 7.5 GiB of runner limits, which leaves healthy headroom for k3s, observability, and burst.
  • 4 replicas would be about 12 CPU / 10 GiB max. That is probably still possible on paper, but it narrows CPU headroom more than necessary on a host that also carries o11y duties.
  • Keeping the 4 existing kai-server runners during the canary already increases total pool capacity materially. There is no need to force immediate parity on day 1.

So the spec should say:

  • Start 1 -> prove it -> scale to 2 -> settle at 3.
  • Consider 4 only after CPU throttling and o11y impact are measured on ser8.

7. Observability

The spec should require ser8-specific identity in both labels and dashboards.

Add stable pod labels such as:

  • app=forgejo-runner
  • runner-plane=build
  • runner-cluster=ser8
  • runner-scope=shared or runner-scope=canary

What to watch:

  • StatefulSet ready replicas
  • init-container registration failures
  • runner container restarts
  • DinD restarts
  • CPU throttling and memory pressure
  • queue/wait time for docker and docker-ser8 jobs
  • image push failures to the kai-server registry

Where visibility should come from:

  • Kubernetes/container metrics for pod health, restarts, CPU, memory
  • Forgejo /metrics for server-side queue and Actions health where available
  • Container logs from register, runner, and dind for registration loops and push failures

Alerting targets:

  • ser8 runner replicas below desired for more than a short window
  • repeated init registration failures
  • repeated DinD crash loops
  • sustained queue growth while ser8 runners are nominally ready

8. Rollback

Rollback should be one action on ser8 only.

Recommended rollback plan:

  • Scale forgejo-runner-ser8 to 0 replicas, or delete only the ser8 manifest.
  • If the pool has already been promoted to docker, remove the docker label before or while scaling down.
  • Leave kai-server deploy:host and kai-server's existing build runners untouched.

This is another reason to keep:

  • a separate ser8 manifest,
  • a separate canary label first,
  • and no coupling to the deploy runner.

Suggested rollout sequence

  1. Add deploy/forgejo-runner-ser8.yml with replicas: 1 and ser8-specific labels.
  2. Register against https://forgejo.coilysiren.me/.
  3. Inject registration secret imperatively on ser8. Do not use tracked secrets.
  4. Give the canary runner docker-ser8 only.
  5. Prove:
    • checkout
    • workflow action download
    • normal docker build
    • image push to the kai-server registry over the chosen remote path
  6. If that holds, add docker to the ser8 runner labels.
  7. Scale to 2 replicas.
  8. After a burn-in period, scale to 3 if queue pressure justifies it.

Risks and open questions

Risks

  • Registry path is the real gating item. Registration to Forgejo is straightforward. Image push from ser8 is the piece that can quietly break shared-label rollout.
  • Single-site dependency remains. Even after runner capacity moves, kai-server still hosts Forgejo and the registry, so a kai-server outage still hurts CI.
  • Insecure-registry drift is worse cross-site than local. The current 192.168.0.194:30500 HTTP shape is tolerable inside the kai-server context, but it is a poor long-term contract for ser8.
  • Immediate shared docker is too risky. Without a canary label, unrelated repos will discover the migration before the operator does.

Open questions the implementation issue should pin down

  • What exact remote registry hostname will ser8 use, and will it be TLS-backed from day 1?
  • Can the runner bootstrap use a bounded registration secret directly, or does the current Forgejo admin-token minting flow need to remain for now?
  • Is the right trust boundary for these runners instance-wide or org-wide? If all intended repos are in one org, org-scoped runners would be a tighter default.

Bottom line

The recommended first spec is:

  • ser8 gets a new infra-owned runner StatefulSet in its own cluster,
  • it registers to https://forgejo.coilysiren.me/,
  • it starts on docker-ser8, not shared docker,
  • it keeps using kai-server's registry via a remote-safe path,
  • secrets are imperative and bounded, because ser8 lacks external-secrets,
  • and rollback is just scale ser8 back to zero.

That gives a real canary without redesigning deploy credentials, while preserving the current kai-server deploy runner and existing build pool until the ser8 path proves itself.

Researched and posted automatically by ward agent advisor --harness codex (ward#179). This is one-shot research, not a carried change - verify before acting on it.

— Codex, via ward agent

### 🔎 ward agent advisor `ward agent advisor --harness codex` ran a one-shot **standard** research pass on this question: > Read the issue title, body, and comment thread below as the research brief. Identify the decisions, options, open questions, and risks, then give a clear advisory recommendation that can be posted back on the issue. --- ## Recommendation The clean first design is **split-plane, same Forgejo**: - Keep `deploy/forgejo-runner-deploy.yml` and the `deploy:host` runner **on kai-server** unchanged. - Add a **separate ser8 runner StatefulSet** for normal `runs-on: docker` build/test capacity. - Treat ser8 as a **second runner pool for the same Forgejo instance**, not as a second deploy plane. - Keep the existing kai-server runners during the canary. Do **not** move or resize the deploy runner in this change. This matches the repo's current shape: - `deploy/forgejo-runner.yml` is today a 4-replica DinD StatefulSet in `forgejo`, registering against `http://forgejo.forgejo.svc.cluster.local` and pushing to the in-cluster registry at `192.168.0.194:30500`. - `deploy/forgejo-runner-deploy.yml` is already a separate `deploy:host` surface with a `deployer` ServiceAccount and is explicitly the wrong thing to move first. - `docs/fleet-reachability.md` confirms **ser8 and kai-server are separate k3s clusters**, and that **ser8 does not have external-secrets** today. ## Decisions ### 1. Registration path **Use `https://forgejo.coilysiren.me/` as `FORGEJO_INSTANCE_URL` for ser8 runners.** Why this is the right default: - The current `forgejo.forgejo.svc.cluster.local` address is cluster-local to kai-server and cannot work from ser8. - The public HTTPS URL is already Forgejo's stable external root URL in this repo's manifests and docs. - It avoids introducing a second reachability dependency just for runner registration. - It uses existing public TLS, so the ser8 runner image and DinD sidecar do not need a custom CA bundle just to talk to Forgejo. TLS/auth implications: - Registration and runner polling happen over normal HTTPS to `forgejo.coilysiren.me`. - No mTLS is needed for this first step. - The secret on ser8 should be a **runner-registration secret**, not a general-purpose Forgejo admin/API token, if Forgejo's registration flow can support it for this pool. - If a broader token is still required for the current init-container pattern, keep it in a Kubernetes `Secret` created out-of-band from SSM or operator input, never in tracked YAML. Why not the tailnet URL first: - It is viable as a fallback, but it adds DNS/routing assumptions between ser8 pods and the Tailscale path. - For the first move, public HTTPS is the lowest-complexity path and matches how users already reach Forgejo. ### 2. Labels and scheduling **Do not put ser8 straight onto the shared `docker` label. Start with a canary label, then promote.** Recommended rollout: 1. First ser8 runner advertises **`docker-ser8` only**. 2. Run a small set of checkout/build/caching/image-push smoke workflows on `docker-ser8`. 3. Once registry push/pull is proven, add the normal **`docker`** label to that same runner pool. 4. Only then scale ser8 replicas up. Reasoning: - If ser8 shares `docker` immediately, any existing image-build workflow can land there before registry reachability is solved. - The current `docker` pool is load-bearing. A bad first cut would turn random CI red instead of creating an isolated canary lane. Promotion target: - After canary, ser8 runners should **also** carry `docker` so capacity becomes automatic. - The long-term steady state can be a mixed pool: kai-server `docker` runners + ser8 `docker` runners + kai-server-only `deploy:host`. ### 3. Registry access **Do not mirror a second registry on ser8, and do not switch package registries in this first move. Keep kai-server's registry as the source of truth, but give ser8 a remote-safe path to it.** Best first design: - Keep the existing registry on kai-server as the image source used by deploys. - Expose that registry to ser8 over a **named remote path** instead of the LAN-only `192.168.0.194:30500` literal. - Prefer a **TLS endpoint** for ser8's DinD daemon, not another insecure plain-HTTP hop if that can be avoided. Why this over the alternatives: - **Mirroring a registry on ser8** creates a new replication/cutover problem and makes deployment image provenance harder to reason about. - **Publishing somewhere else** such as Forgejo packages or GHCR is a broader cross-repo rollout. It changes app image refs and pull-secret handling across downstream repos, which is larger than this spec needs. - **Keeping `192.168.0.194:30500`** only works from kai-server/LAN-adjacent paths. It is the wrong address from ser8. Failure modes to call out: - The registry remains physically hosted on kai-server, so ser8 builds still depend on kai-server reachability. - If the remote registry path is still plain HTTP plus `--insecure-registry`, that remains acceptable only as a temporary canary compromise. It is brittle and not a good permanent cross-site shape. - If the registry path is fronted with TLS, certificate trust and hostname stability get much better, but push latency becomes dependent on the WAN/tailnet hop. Practical recommendation: - **Canary phase:** a remote path to the existing kai-server registry is enough. - **Permanent phase:** prefer a TLS name for the registry so ser8 does not need to normalize insecure-registry behavior forever. ### 4. Secrets Because ser8 lacks external-secrets, the secret flow should be **bounded and imperative**, mirroring the existing ser8 pattern used elsewhere in this repo. Recommended flow: - Create the ser8 runner secret as a normal Kubernetes `Secret` at apply time. - Materialize it from SSM or operator input on the control host, not from tracked YAML. - Store the **minimum credential needed to register runners**. Preference order: 1. **Best:** a runner-registration secret scoped to this runner pool. 2. **Second-best:** a dedicated Forgejo token used only to mint registration tokens for this pool. 3. **Avoid:** reusing a broad admin token when a narrower token works. Operational note: - The repo already uses this style for ser8-specific cluster rollout in `docs/fleet-reachability.md`: resolve from SSM on the control host, then create the in-cluster secret imperatively. - The same pattern fits here. ### 5. Namespace and manifest ownership **Keep this infra-owned and manifest-owned in this repo.** Recommended ownership: - Add a dedicated manifest such as **`deploy/forgejo-runner-ser8.yml`**. - Keep the namespace name **`forgejo`** on ser8 as well. Why: - The workload is still cluster infrastructure, not app-specific deploy config. - A separate file avoids accidental coupling to the kai-server-only assumptions in `deploy/forgejo-runner.yml`. - Reusing the `forgejo` namespace keeps operational queries symmetric across clusters. Rollout method: - First implementation should be a **human-run cluster apply**, not something hidden inside `ansible-sync`. - If a later ansible wrapper is added, it should look like `fleet-reachability`: explicit, per-cluster, and confirm-before-apply. ### 6. Resource sizing Current kai-server per-replica limits are effectively: - runner: **1 CPU / 512 MiB** - DinD: **2 CPU / 2 GiB** - total per replica: about **3 CPU / 2.5 GiB limit** For ser8 (`57 GiB RAM`, `16 cores`), the safest recommendation is: - **Canary:** `replicas: 1`, same limits as kai-server. - **First scale-up:** `replicas: 2` after checkout/build/image-push smoke passes. - **Steady-state target:** `replicas: 3` on ser8 before considering 4. Why 3 is the right initial target: - 3 replicas cap at about **9 CPU / 7.5 GiB** of runner limits, which leaves healthy headroom for k3s, observability, and burst. - 4 replicas would be about **12 CPU / 10 GiB** max. That is probably still possible on paper, but it narrows CPU headroom more than necessary on a host that also carries o11y duties. - Keeping the 4 existing kai-server runners during the canary already increases total pool capacity materially. There is no need to force immediate parity on day 1. So the spec should say: - **Start 1 -> prove it -> scale to 2 -> settle at 3**. - Consider 4 only after CPU throttling and o11y impact are measured on ser8. ### 7. Observability The spec should require **ser8-specific identity** in both labels and dashboards. Add stable pod labels such as: - `app=forgejo-runner` - `runner-plane=build` - `runner-cluster=ser8` - `runner-scope=shared` or `runner-scope=canary` What to watch: - StatefulSet ready replicas - init-container registration failures - runner container restarts - DinD restarts - CPU throttling and memory pressure - queue/wait time for `docker` and `docker-ser8` jobs - image push failures to the kai-server registry Where visibility should come from: - **Kubernetes/container metrics** for pod health, restarts, CPU, memory - **Forgejo `/metrics`** for server-side queue and Actions health where available - **Container logs** from `register`, `runner`, and `dind` for registration loops and push failures Alerting targets: - ser8 runner replicas below desired for more than a short window - repeated init registration failures - repeated DinD crash loops - sustained queue growth while ser8 runners are nominally ready ### 8. Rollback Rollback should be **one action on ser8 only**. Recommended rollback plan: - Scale `forgejo-runner-ser8` to `0` replicas, or delete only the ser8 manifest. - If the pool has already been promoted to `docker`, remove the `docker` label before or while scaling down. - Leave kai-server `deploy:host` and kai-server's existing build runners untouched. This is another reason to keep: - a separate ser8 manifest, - a separate canary label first, - and no coupling to the deploy runner. ## Suggested rollout sequence 1. Add `deploy/forgejo-runner-ser8.yml` with `replicas: 1` and ser8-specific labels. 2. Register against `https://forgejo.coilysiren.me/`. 3. Inject registration secret imperatively on ser8. Do not use tracked secrets. 4. Give the canary runner **`docker-ser8` only**. 5. Prove: - checkout - workflow action download - normal docker build - image push to the kai-server registry over the chosen remote path 6. If that holds, add `docker` to the ser8 runner labels. 7. Scale to 2 replicas. 8. After a burn-in period, scale to 3 if queue pressure justifies it. ## Risks and open questions ### Risks - **Registry path is the real gating item.** Registration to Forgejo is straightforward. Image push from ser8 is the piece that can quietly break shared-label rollout. - **Single-site dependency remains.** Even after runner capacity moves, kai-server still hosts Forgejo and the registry, so a kai-server outage still hurts CI. - **Insecure-registry drift is worse cross-site than local.** The current `192.168.0.194:30500` HTTP shape is tolerable inside the kai-server context, but it is a poor long-term contract for ser8. - **Immediate shared `docker` is too risky.** Without a canary label, unrelated repos will discover the migration before the operator does. ### Open questions the implementation issue should pin down - What exact remote registry hostname will ser8 use, and will it be TLS-backed from day 1? - Can the runner bootstrap use a bounded registration secret directly, or does the current Forgejo admin-token minting flow need to remain for now? - Is the right trust boundary for these runners **instance-wide** or **org-wide**? If all intended repos are in one org, org-scoped runners would be a tighter default. ## Bottom line The recommended first spec is: - **ser8 gets a new infra-owned runner StatefulSet** in its own cluster, - it **registers to `https://forgejo.coilysiren.me/`**, - it starts on **`docker-ser8`**, not shared `docker`, - it keeps using **kai-server's registry** via a remote-safe path, - secrets are **imperative and bounded**, because ser8 lacks external-secrets, - and rollback is just **scale ser8 back to zero**. That gives a real canary without redesigning deploy credentials, while preserving the current kai-server deploy runner and existing build pool until the ser8 path proves itself. --- Researched and posted automatically by `ward agent advisor --harness codex` (ward#179). This is one-shot research, not a carried change - verify before acting on it. <!-- ward-agent-reply --> <!-- ward-agent-signature --> — Codex, via `ward agent`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-515
issue-486
issue-529
issue-500
issue-483
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
consult
A human decision, design, or external action must happen first. Fail-closed default for unlabeled/low-confidence.

  
headless
Agent can take it from open issue to merged change with no human in the loop. The auto-burndown queue.

  
interactive
Agent does the work but pauses at a human checkpoint mid-flight (design choice, destructive step, human-only verification).

  
P0
now / blocking / urgent

  
P1
next / important

  
P2
backlog / will do

  
P3
someday / low

  
P4
icebox / frozen / reopen if it becomes real

No labels
burndown-2026-06
coherence-core
consult
headless
interactive
P0
P1
P2
P3
P4
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coilyco-flight-deck/infrastructure#507
No description provided.