coilyco-flight-deck/infrastructure

Watch

Operator cleanup: retire server-native agent services #667

New issue

Closed

opened 2026-07-28 18:26:57 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-07-28 18:26:57 +00:00

Member

Operator boundary

Commits c311353 and 10753b4 remove the tracked server-native agent surfaces and establish the warded-only server profile. The engineer role did not mutate live hosts.

Operator action

Kai or an Ops run follows the committed server-native agent retirement runbook once on kai-server and ser8. The runbook owns the exact Ansible and systemd commands.

Evidence to post

The Ansible server-profile converge succeeds for kai-server and ser8.
Every retired unit is absent or inactive.
~/.mcporter/mcporter.json is a regular file on both hosts.
Ward launches a normal isolated session after cleanup.

The operator stops and posts the failing command and output if any check fails.

## Operator boundary Commits `c311353` and `10753b4` remove the tracked server-native agent surfaces and establish the warded-only server profile. The engineer role did not mutate live hosts. ## Operator action Kai or an Ops run follows the committed [server-native agent retirement runbook](https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/src/branch/main/docs/systemd-native-agent-retirement.md) once on kai-server and ser8. The runbook owns the exact Ansible and systemd commands. ## Evidence to post * The Ansible server-profile converge succeeds for kai-server and ser8. * Every retired unit is absent or inactive. * `~/.mcporter/mcporter.json` is a regular file on both hosts. * Ward launches a normal isolated session after cleanup. The operator stops and posts the failing command and output if any check fails.

coilyco-ops added the

autonomy

live-collab

label

2026-07-28 18:27:03 +00:00

coilyco-ops referenced this issue

2026-07-28 18:27:10 +00:00

Preserve retired agentic-os pin convergence implementation #666

coilyco-ops commented

2026-07-28 18:43:03 +00:00

Author

Member

Commit 0bc67b3 extends the same operator pass with the staggered host-convergence timers.

kai-server installs an hourly fleet-converge.timer at :00 targeting only kai-server.
ser8 installs an hourly timer at :30 targeting ser8,kai-server.
Both use the fleet inventory for self-targeting, apply on every scheduled tick, disable jitter and missed-run catch-up, and cap the Ward command at 25 minutes.
Both hosts receive the Telegram cache role. A failed convergence alerts the red channel and a newly applied commit reports quietly to green.

The committed retirement runbook now includes timer and lingering verification. In addition to the original cleanup evidence, the Ops run should post the next timer timestamp from each host, confirm Linger=yes on both, and post the last fleet-converge.service result. The engineer role did not apply this to either live host.

Commit `0bc67b3` extends the same operator pass with the staggered host-convergence timers. * kai-server installs an hourly `fleet-converge.timer` at `:00` targeting only `kai-server`. * ser8 installs an hourly timer at `:30` targeting `ser8,kai-server`. * Both use the fleet inventory for self-targeting, apply on every scheduled tick, disable jitter and missed-run catch-up, and cap the Ward command at 25 minutes. * Both hosts receive the Telegram cache role. A failed convergence alerts the red channel and a newly applied commit reports quietly to green. The committed retirement runbook now includes timer and lingering verification. In addition to the original cleanup evidence, the Ops run should post the next timer timestamp from each host, confirm `Linger=yes` on both, and post the last `fleet-converge.service` result. The engineer role did not apply this to either live host.

coilyco-ops commented

2026-07-28 20:38:17 +00:00

Author

Member

Ops applied and verified the ser8 half on 2026-07-28.

Completed live state:

ser8 server-profile convergence passes locally with the controller connection.
The retired agentic-os-pin-converge service, timer, and 331-line script are absent on ser8.
Linger=yes, Ward reports v0.630.0, and ~/.mcporter/mcporter.json is a regular file on ser8.
The retired o2r formula and coilyco-flight-deck/otel-a2a-relay-cli tap are absent on both ser8 and kai-server.
Bundled Forgejo Git and Docker helpers on ser8 return the current SSM credentials and contain no retired SSM path.
The service-owned infrastructure runtime checkout is clean, tracks origin/main, and reached canonical commit 8a4c116 without touching ser8s dirty operator checkout.

Landed fixes include controller-local self-targeting, SSH config isolation, bundled credential-helper rollout, in-process registry read-token reminting, Homebrew formula-before-tap retirement ordering, and a Ward-auditable service-owned converge checkout. Infrastructure lint and secret-scan workflows passed for 8a4c116. All agentic-os workflows passed for credential commit 18eb13de.

Current wall:

The ser8 service completed its local leg and reached kai-server over SSH.
kai-server then failed at apt-baseline with Missing sudo password.
sudo -n -l confirms that user kai has only command-scoped NOPASSWD grants, not unattended Ansible become. Neither host has an OS-level ops account.
Automatic sudoers rollout is still explicitly out of scope in docs/FEATURES.md, so Ops did not widen privilege without Kai choosing that security boundary.

Safe rollback:

Ops disabled and stopped only ser8s fleet-converge.timer to prevent hourly failure alerts.
The timer and service definitions remain installed. The service is inactive with failed state cleared.

Next human decision: authorize a one-time attended kai-server bootstrap that grants the convergence controller passwordless become, or direct the scheduled controller to run an explicitly unprivileged role set. After that decision, Ops can re-enable ser8 at :30 and post one successful two-host service result here. This issue remains open.

Ops applied and verified the ser8 half on 2026-07-28. Completed live state: * ser8 server-profile convergence passes locally with the controller connection. * The retired `agentic-os-pin-converge` service, timer, and 331-line script are absent on ser8. * `Linger=yes`, Ward reports `v0.630.0`, and `~/.mcporter/mcporter.json` is a regular file on ser8. * The retired `o2r` formula and `coilyco-flight-deck/otel-a2a-relay-cli` tap are absent on both ser8 and kai-server. * Bundled Forgejo Git and Docker helpers on ser8 return the current SSM credentials and contain no retired SSM path. * The service-owned infrastructure runtime checkout is clean, tracks `origin/main`, and reached canonical commit `8a4c116` without touching ser8s dirty operator checkout. Landed fixes include controller-local self-targeting, SSH config isolation, bundled credential-helper rollout, in-process registry read-token reminting, Homebrew formula-before-tap retirement ordering, and a Ward-auditable service-owned converge checkout. Infrastructure lint and secret-scan workflows passed for `8a4c116`. All agentic-os workflows passed for credential commit `18eb13de`. Current wall: * The ser8 service completed its local leg and reached kai-server over SSH. * kai-server then failed at `apt-baseline` with `Missing sudo password`. * `sudo -n -l` confirms that user `kai` has only command-scoped NOPASSWD grants, not unattended Ansible become. Neither host has an OS-level `ops` account. * Automatic sudoers rollout is still explicitly out of scope in `docs/FEATURES.md`, so Ops did not widen privilege without Kai choosing that security boundary. Safe rollback: * Ops disabled and stopped only ser8s `fleet-converge.timer` to prevent hourly failure alerts. * The timer and service definitions remain installed. The service is inactive with failed state cleared. Next human decision: authorize a one-time attended kai-server bootstrap that grants the convergence controller passwordless become, or direct the scheduled controller to run an explicitly unprivileged role set. After that decision, Ops can re-enable ser8 at `:30` and post one successful two-host service result here. This issue remains open.

coilyco-ops commented

2026-07-28 21:31:10 +00:00

Author

Member

Ops completed the ser8 privileged-drift alert path on 2026-07-28.

Landed behavior:

ansible/playbooks/sync.yml now owns separate scheduled and privileged role inventories.
The scheduled set is observer-only: Telegram cache, controller and observer timers, and marker reporting. General host convergence stays attended even when password-free.
Full untagged applies write a mode-0600 per-host marker only after every earlier role succeeds. Tagged and check runs cannot refresh it.
Ser8 compares those markers with a Git-tree fingerprint of privileged roles plus server-fleet inputs. Mac and Windows inventory changes do not page the servers.
Missing or stale targets alert the Telegram red channel every tick while the scheduled apply itself exits successfully.
No sudoers policy was widened.

Live evidence:

Ser8 fleet-converge.timer is enabled at :30 with the next hourly tick scheduled.
The manual service proof returned Result=success and ExecMainStatus=0.
The controller report classified ser8=current and kai-server=pending.
Ser8s marker is mode 0600, and all three Telegram cache files are present and non-empty.
The service journal emitted [fleet-converge: privileged apply pending] targets=kai-server followed by ward exec ansible-sync apply hosts=kai-server ask_pass.
Forgejo lint and secret-scan workflows passed for the final commit 8eb845e.

The initial live pass exposed two roles that were password-free but unsuitable for hourly control. dev-harness hit private-registry authorization drift, and Homebrew lost the long-running SSH channel. Ops removed both, along with the other general host-convergence roles, from the scheduled set instead of masking those failures.

Kai runs ward exec ansible-sync apply hosts=kai-server ask_pass when ready. The next ser8 tick will observe the new marker, clear its owned pending state, and send the green recovery line. This issue remains open until that attended kai-server apply and recovery tick are verified.

Ops completed the ser8 privileged-drift alert path on 2026-07-28. Landed behavior: * `ansible/playbooks/sync.yml` now owns separate `scheduled` and `privileged` role inventories. * The scheduled set is observer-only: Telegram cache, controller and observer timers, and marker reporting. General host convergence stays attended even when password-free. * Full untagged applies write a mode-0600 per-host marker only after every earlier role succeeds. Tagged and check runs cannot refresh it. * Ser8 compares those markers with a Git-tree fingerprint of privileged roles plus server-fleet inputs. Mac and Windows inventory changes do not page the servers. * Missing or stale targets alert the Telegram red channel every tick while the scheduled apply itself exits successfully. * No sudoers policy was widened. Live evidence: * Ser8 `fleet-converge.timer` is enabled at `:30` with the next hourly tick scheduled. * The manual service proof returned `Result=success` and `ExecMainStatus=0`. * The controller report classified `ser8=current` and `kai-server=pending`. * Ser8s marker is mode `0600`, and all three Telegram cache files are present and non-empty. * The service journal emitted `[fleet-converge: privileged apply pending] targets=kai-server` followed by `ward exec ansible-sync apply hosts=kai-server ask_pass`. * Forgejo lint and secret-scan workflows passed for the final commit `8eb845e`. The initial live pass exposed two roles that were password-free but unsuitable for hourly control. `dev-harness` hit private-registry authorization drift, and Homebrew lost the long-running SSH channel. Ops removed both, along with the other general host-convergence roles, from the scheduled set instead of masking those failures. Kai runs `ward exec ansible-sync apply hosts=kai-server ask_pass` when ready. The next ser8 tick will observe the new marker, clear its owned pending state, and send the green recovery line. This issue remains open until that attended kai-server apply and recovery tick are verified.

coilyco-ops added the

priority

label

2026-07-29 03:46:38 +00:00

coilyco-ops closed this issue

2026-07-29 06:04:37 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/infrastructure#667

No description provided.

Rows
Columns