feat(evalkit): wire the transport and the composed-prompt producer #264

Merged

coilysiren merged 1 commit from adjacency/262-role-adjacents into main

2026-08-10 19:57:19 +00:00

coilysiren commented

2026-08-10 19:56:24 +00:00

Owner

evalkit.run took a --prompts directory of composed .md that nothing
produced. scripts/eval-prompts.sh composes one bundle per role and writes
its compiled delivery there, which is the actual system prompt a role
receives. Frontier is the only tier every role supports, and model tier
does not change selected context.

scripts/eval-smoke.sh sends one live request so a broken transport
surfaces before a board run rather than during one.

Three corrections from exercising the real path for the first time:

Agent Proxy authenticates by network position on the tailnet, so
--api-key is now optional and the Authorization header is omitted when
it is empty. Base URL and model now default to the live proxy and
evaluation/deepseek-v4-flash.

The subject is a reasoning model and returns reasoning_content beside
content. It is preserved as evidence, never graded, and never counted
against the word cap, where observed reasoning ran up to 1761 words for
a 21-word answer.

The local venv had partially installed pyyaml and httpx: the packages
registered as present while their Python sources were missing, so both
CLIs raised on import while every gate passed green. Nothing exercised
the serialization path. test_io.py now round-trips boards, grades, and
responses through disk, and asserts the two attributes the CLIs call.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: Kai Siren coilysiren@gmail.com
Co-authored-by: Claude noreply@anthropic.com

evalkit.run took a --prompts directory of composed <role>.md that nothing produced. scripts/eval-prompts.sh composes one bundle per role and writes its compiled delivery there, which is the actual system prompt a role receives. Frontier is the only tier every role supports, and model tier does not change selected context. scripts/eval-smoke.sh sends one live request so a broken transport surfaces before a board run rather than during one. Three corrections from exercising the real path for the first time: Agent Proxy authenticates by network position on the tailnet, so --api-key is now optional and the Authorization header is omitted when it is empty. Base URL and model now default to the live proxy and evaluation/deepseek-v4-flash. The subject is a reasoning model and returns reasoning_content beside content. It is preserved as evidence, never graded, and never counted against the word cap, where observed reasoning ran up to 1761 words for a 21-word answer. The local venv had partially installed pyyaml and httpx: the packages registered as present while their Python sources were missing, so both CLIs raised on import while every gate passed green. Nothing exercised the serialization path. test_io.py now round-trips boards, grades, and responses through disk, and asserts the two attributes the CLIs call. Refs #262. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kai Siren <coilysiren@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

coilysiren added 1 commit

2026-08-10 19:56:24 +00:00

feat(evalkit): wire the transport and the composed-prompt producer 8a7629d7bc

evalkit.run took a --prompts directory of composed <role>.md that nothing
produced. scripts/eval-prompts.sh composes one bundle per role and writes
its compiled delivery there, which is the actual system prompt a role
receives. Frontier is the only tier every role supports, and model tier
does not change selected context.

scripts/eval-smoke.sh sends one live request so a broken transport
surfaces before a board run rather than during one.

Three corrections from exercising the real path for the first time:

Agent Proxy authenticates by network position on the tailnet, so
--api-key is now optional and the Authorization header is omitted when
it is empty. Base URL and model now default to the live proxy and
evaluation/deepseek-v4-flash.

The subject is a reasoning model and returns reasoning_content beside
content. It is preserved as evidence, never graded, and never counted
against the word cap, where observed reasoning ran up to 1761 words for
a 21-word answer.

The local venv had partially installed pyyaml and httpx: the packages
registered as present while their Python sources were missing, so both
CLIs raised on import while every gate passed green. Nothing exercised
the serialization path. test_io.py now round-trips boards, grades, and
responses through disk, and asserts the two attributes the CLIs call.

Refs #262.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kai Siren <coilysiren@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>

coilysiren merged commit 0c8b733eb1 into main

2026-08-10 19:57:19 +00:00

coilysiren referenced this pull request from a commit

2026-08-10 19:57:19 +00:00

Merge pull request 'feat(evalkit): wire the transport and the composed-prompt producer' (#264) from adjacency/262-role-adjacents into main

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agent-compose!264

No description provided.

Rows
Columns