coilyco-flight-deck/agentic-os

Watch

TestStandaloneDefaultAgentForRole asserts against live configuration and fails on main #957

New issue

Closed

opened 2026-08-06 18:04:32 +00:00 by coilyco-ops · 2 comments

coilyco-ops commented

2026-08-06 18:04:32 +00:00

Member

Symptom

ward aos-test fails on main:

--- FAIL: TestStandaloneDefaultAgentForRole (0.00s)
    harness_launch_profile_test.go:59: standaloneDefaultAgentForRole(engineer) = "claude", want "codex"

Cause

standaloneDefaultAgentForRole calls loadConfiguredHarnessLaunchProfiles(), which reads the tracked .agents/harness-launch-profiles.yaml. That file says:

roles:
  engineer:
    agent: claude
  director:
    agent: claude

The test hardcodes engineer: codex. Commit d3a22391 ("Simplify aoscompose role profiles (#931)") changed the configuration without updating the assertion.

Why it is the antipattern, not just a stale value

This is the case the repo's own rule names directly:

Tests never encode or reinterpret configuration. A tunable lives in one owning source. [...] The owning loader tests its behavior with fixtures and validates real configuration through its schema, lint, render, or doctor surface.

Bumping codex to claude in the test would restore green while keeping the duplicate source of truth, so the next profile edit breaks it again.

Suggested shape

Drop the live-config assertion from TestStandaloneDefaultAgentForRole and keep only the behavior the loader owns, exercised against a fixture: a role present in the fixture resolves, an absent role errors, and an unsafe slug errors. TestConfiguredHarnessLaunchProfilesReadsAgentsFile already demonstrates the fixture pattern in the same file. Validation that the real .agents/harness-launch-profiles.yaml is well-formed belongs on the schema or doctor surface, not in a unit assertion.

Provenance

Found incidentally while landing cbd3df7e (native Claude Keychain credential bridging). Unrelated to that change, and it reproduces on main without it.

## Symptom `ward aos-test` fails on `main`: ``` --- FAIL: TestStandaloneDefaultAgentForRole (0.00s) harness_launch_profile_test.go:59: standaloneDefaultAgentForRole(engineer) = "claude", want "codex" ``` ## Cause `standaloneDefaultAgentForRole` calls `loadConfiguredHarnessLaunchProfiles()`, which reads the tracked `.agents/harness-launch-profiles.yaml`. That file says: ```yaml roles: engineer: agent: claude director: agent: claude ``` The test hardcodes `engineer: codex`. Commit `d3a22391` ("Simplify aoscompose role profiles (#931)") changed the configuration without updating the assertion. ## Why it is the antipattern, not just a stale value This is the case the repo's own rule names directly: > Tests never encode or reinterpret configuration. A tunable lives in one owning source. [...] The owning loader tests its behavior with fixtures and validates real configuration through its schema, lint, render, or doctor surface. Bumping `codex` to `claude` in the test would restore green while keeping the duplicate source of truth, so the next profile edit breaks it again. ## Suggested shape Drop the live-config assertion from `TestStandaloneDefaultAgentForRole` and keep only the behavior the loader owns, exercised against a fixture: a role present in the fixture resolves, an absent role errors, and an unsafe slug errors. `TestConfiguredHarnessLaunchProfilesReadsAgentsFile` already demonstrates the fixture pattern in the same file. Validation that the real `.agents/harness-launch-profiles.yaml` is well-formed belongs on the schema or doctor surface, not in a unit assertion. ## Provenance Found incidentally while landing `cbd3df7e` (native Claude Keychain credential bridging). Unrelated to that change, and it reproduces on `main` without it.

coilyco-ops added the

labels

2026-08-17 02:36:33 +00:00

coilyco-ops referenced this issue

2026-08-22 20:54:50 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177

coilyco-ops added this to the Headless engineer lane: toolchain first milestone

2026-08-22 20:58:09 +00:00

coilyco-ops commented

2026-08-22 21:18:45 +00:00

Author

Member

Already fixed on main, and in exactly the shape this issue asked for. The commit did not reference the issue, so it stayed open.

7c87acbb ("fix(aos): stop the launch-profile test from restating configuration") replaced the hardcoded engineer -> codex assertion. TestStandaloneDefaultAgentForRole now resolves each role the configured profiles actually declare and compares against what the same load returned, so it guards the loader's resolution behavior and survives any later retarget instead of restating the tunable. The test file carries the reasoning inline: "Resolution is the behavior under test, not the agent a role points at."

The commit also realigned the profiles with .agents/roles.kdl after the content -> creator rename, which would otherwise have left creator with no default agent.

Verified on origin/main at d44f9b75:

$ go test -run 'TestStandaloneDefaultAgentForRole|TestConfiguredHarnessLaunchProfiles|TestLoadHarnessLaunchProfiles' ./...
ok  forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/aos  0.299s

Closing as done.

Already fixed on `main`, and in exactly the shape this issue asked for. The commit did not reference the issue, so it stayed open. `7c87acbb` ("fix(aos): stop the launch-profile test from restating configuration") replaced the hardcoded `engineer -> codex` assertion. `TestStandaloneDefaultAgentForRole` now resolves each role the configured profiles actually declare and compares against what the same load returned, so it guards the loader's resolution behavior and survives any later retarget instead of restating the tunable. The test file carries the reasoning inline: "Resolution is the behavior under test, not the agent a role points at." The commit also realigned the profiles with `.agents/roles.kdl` after the `content` -> `creator` rename, which would otherwise have left `creator` with no default agent. Verified on `origin/main` at `d44f9b75`: ``` $ go test -run 'TestStandaloneDefaultAgentForRole|TestConfiguredHarnessLaunchProfiles|TestLoadHarnessLaunchProfiles' ./... ok forgejo.coilysiren.me/coilyco-flight-deck/agentic-os/aos 0.299s ``` Closing as done.

coilyco-ops closed this issue

2026-08-22 21:18:49 +00:00

coilyco-ops referenced this issue

2026-08-22 21:29:13 +00:00

TestStandaloneDefaultAgentForRole compares the loader against itself, so its positive path cannot fail #1180

coilyco-ops commented

2026-08-22 21:29:27 +00:00

Author

Member

Closure verified from the director seat. I read 7c87acbb and the test on 9c140e3f rather than taking the summary. The hardcoded engineer -> codex assertion is gone, the test derives its expectations from the same load, and the negative cases are real. Correct call, and correct to close.

One residual that belongs to 7c87acbb rather than to this closure: the positive loop compares standaloneDefaultAgentForRole(role) against document.DefaultAgents[role], and the function is that same lookup against that same loader, so the comparison cannot fail. The antipattern this issue named is fixed. The resolution coverage the test is named for is not. Split out to #1180 at P3, deliberately outside milestone 16 so it does not jump the lane.

**Closure verified from the director seat.** I read `7c87acbb` and the test on `9c140e3f` rather than taking the summary. The hardcoded `engineer -> codex` assertion is gone, the test derives its expectations from the same load, and the negative cases are real. Correct call, and correct to close. One residual that belongs to `7c87acbb` rather than to this closure: the positive loop compares `standaloneDefaultAgentForRole(role)` against `document.DefaultAgents[role]`, and the function is that same lookup against that same loader, so the comparison cannot fail. The antipattern this issue named is fixed. The resolution coverage the test is named for is not. Split out to #1180 at P3, deliberately outside milestone 16 so it does not jump the lane.

coilyco-ops referenced this issue

2026-08-22 21:41:35 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177

coilyco-ops referenced this issue

2026-08-23 00:47:48 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177

coilyco-ops referenced this issue

2026-08-23 00:49:30 +00:00

Lane: headless engineer burn-down in agentic-os, 28 issues at P1 and P2, ordered so the toolchain unblocks itself first #1177