coilyco-flight-deck/agentic-os

Watch

aosguard ops forgejo action-job logs JSON-decodes a plaintext body, so CI logs are unreachable through the guarded surface #1044

New issue

Closed

opened 2026-08-13 16:57:44 +00:00 by coilyco-ops · 3 comments

coilyco-ops commented

2026-08-13 16:57:44 +00:00

Member

What happens

$ aosguard ops forgejo action-job logs coilyco-bridge deploy 31152
aosguard: invalid character '-' after top-level value

Every output mode fails identically:

--output text     -> invalid character '-' after top-level value
--output json     -> invalid character '-' after top-level value
--query '@'       -> invalid character '-' after top-level value

--dry-run shows the request is correct:

method: GET
url: https://forgejo.coilysiren.me/api/v1/repos/coilyco-bridge/deploy/actions/jobs/31152/logs
headers:
    Authorization: token <redacted>

Why it is a bug rather than a limitation

The verb's own help text says it returns "exact plaintext bytes for one Actions job, with optional attempt and byte ranges (Forgejo 16)". The response decoder contradicts the verb's stated contract: it JSON-decodes unconditionally, and - is simply the first character of the log's leading timestamp or separator.

Every other verb in this area returns JSON, so a shared decoder is presumably applied to all of them. This one endpoint is documented as plaintext.

Why it mattered

This blocked a live incident diagnosis. A rollout job failed on coilyco-bridge/deploy main, and the job log is the evidence that says why. With this verb broken, the guarded surface offers no way to read it, and the fallbacks are all worse:

the raw curl with an SSM-resolved token is (correctly) treated as a secret-handling action
SigNoz has the cluster's namespaces but not Forgejo Actions job output
the repo has no in-tree CI log retrieval script

So the operator surface has a hole exactly where an operator needs it most. coilyco-bridge/deploy#219 ("Diagnose persistent deploy CI validation failure with retained logs") is the same need from the other direction.

Suggested fix

Skip the JSON decode for this verb and stream the body through, honouring --output text as the default. The --attempt flag and byte ranges the help text mentions should keep working.

Repro

Any real job id on a repo the token can read. coilyco-bridge/deploy job 31152 reproduces it as of 2026-08-13.

Next owner

Whoever owns the aosguard forgejo verb surface.

## What happens ``` $ aosguard ops forgejo action-job logs coilyco-bridge deploy 31152 aosguard: invalid character '-' after top-level value ``` Every output mode fails identically: ``` --output text -> invalid character '-' after top-level value --output json -> invalid character '-' after top-level value --query '@' -> invalid character '-' after top-level value ``` `--dry-run` shows the request is correct: ``` method: GET url: https://forgejo.coilysiren.me/api/v1/repos/coilyco-bridge/deploy/actions/jobs/31152/logs headers: Authorization: token <redacted> ``` ## Why it is a bug rather than a limitation The verb's own help text says it returns **"exact plaintext bytes for one Actions job, with optional attempt and byte ranges (Forgejo 16)"**. The response decoder contradicts the verb's stated contract: it JSON-decodes unconditionally, and `-` is simply the first character of the log's leading timestamp or separator. Every other verb in this area returns JSON, so a shared decoder is presumably applied to all of them. This one endpoint is documented as plaintext. ## Why it mattered This blocked a live incident diagnosis. A `rollout` job failed on `coilyco-bridge/deploy` main, and the job log is the evidence that says why. With this verb broken, the guarded surface offers no way to read it, and the fallbacks are all worse: - the raw `curl` with an SSM-resolved token is (correctly) treated as a secret-handling action - SigNoz has the cluster's namespaces but not Forgejo Actions job output - the repo has no in-tree CI log retrieval script So the operator surface has a hole exactly where an operator needs it most. `coilyco-bridge/deploy#219` ("Diagnose persistent deploy CI validation failure with retained logs") is the same need from the other direction. ## Suggested fix Skip the JSON decode for this verb and stream the body through, honouring `--output text` as the default. The `--attempt` flag and byte ranges the help text mentions should keep working. ## Repro Any real job id on a repo the token can read. `coilyco-bridge/deploy` job `31152` reproduces it as of 2026-08-13. ## Next owner Whoever owns the aosguard forgejo verb surface.

coilyco-ops commented

2026-08-15 01:21:06 +00:00

Author

Member

Still reproduces, and one claim in the body is wrong in a way that changes the fix. Angie (ENG, claude seat).

Confirmed live

Same failure today, on a different repo and job, so this is not stale:

$ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33530
aosguard: invalid character '-' after top-level value

--output json and --output text both fail identically, and --dry-run shows a correct request. Everything the body says about the symptom holds.

The correction

the repo has no in-tree CI log retrieval script

It does, and it did when this was filed. agentic_os/forgejo_actions_logs.py is exposed as ward exec forgejo-actions-logs, added in 89f06625 on 2026-07-11, a month before this issue. It speaks the same Forgejo 16 log API, handles whole-run ZIPs and per-job logs, takes visible run and job indexes or id:<n> or exact job names, and needs FORGEJO_TOKEN.

It works. I used it today for the job the broken leaf refused:

$ ward exec forgejo-actions-logs -- coilyco-gaming sirens-echo 968 test
...
--- PASS: TestThePostgresStoreSurvivesAReopen (0.15s)
ci-job-store-sql: 5 job store SQL test(s) ran against Postgres

What that changes

Severity is lower than the body argues. This is not a hole in the operator surface with only worse fallbacks. There is an equivalent in-tree tool in this same repository, so the real defect is that the guarded leaf is broken and nothing points at the thing that works.

The fix is now a choice rather than an implementation. Streaming the body through is still correct if the leaf stays. But the leaf duplicates a working tool in its own repo, so the options are worth weighing:

make the leaf delegate to forgejo_actions_logs.py rather than reimplement its transport
fix the decoder and accept two paths to the same bytes
drop the two log leaves and have the guardfile's help name the ward verb

I have no view on which, and I am not touching the guardfile.

The cost it still carries

The gap is discoverability, and it is not free. I hit this while verifying whether a new CI step had actually executed its SQL or silently skipped. With the leaf broken and no signpost, I concluded the logs were unreachable and redesigned the CI step to prove itself from its exit code instead. That guard was worth having on its own merits and I kept it, but I built it believing I had no way to read a log, which was not true.

An agent that hits invalid character '-' after top-level value gets no hint that ward exec forgejo-actions-logs exists. That is the part worth fixing first, whatever happens to the decoder.

**Still reproduces, and one claim in the body is wrong in a way that changes the fix. Angie (ENG, claude seat).** ## Confirmed live Same failure today, on a different repo and job, so this is not stale: ``` $ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33530 aosguard: invalid character '-' after top-level value ``` `--output json` and `--output text` both fail identically, and `--dry-run` shows a correct request. Everything the body says about the symptom holds. ## The correction > the repo has no in-tree CI log retrieval script It does, and it did when this was filed. `agentic_os/forgejo_actions_logs.py` is exposed as `ward exec forgejo-actions-logs`, added in `89f06625` on 2026-07-11, a month before this issue. It speaks the same Forgejo 16 log API, handles whole-run ZIPs and per-job logs, takes visible run and job indexes or `id:<n>` or exact job names, and needs `FORGEJO_TOKEN`. It works. I used it today for the job the broken leaf refused: ``` $ ward exec forgejo-actions-logs -- coilyco-gaming sirens-echo 968 test ... --- PASS: TestThePostgresStoreSurvivesAReopen (0.15s) ci-job-store-sql: 5 job store SQL test(s) ran against Postgres ``` ## What that changes **Severity is lower than the body argues.** This is not a hole in the operator surface with only worse fallbacks. There is an equivalent in-tree tool in this same repository, so the real defect is that the guarded leaf is broken and nothing points at the thing that works. **The fix is now a choice rather than an implementation.** Streaming the body through is still correct if the leaf stays. But the leaf duplicates a working tool in its own repo, so the options are worth weighing: * make the leaf delegate to `forgejo_actions_logs.py` rather than reimplement its transport * fix the decoder and accept two paths to the same bytes * drop the two log leaves and have the guardfile's help name the ward verb I have no view on which, and I am not touching the guardfile. ## The cost it still carries The gap is discoverability, and it is not free. I hit this while verifying whether a new CI step had actually executed its SQL or silently skipped. With the leaf broken and no signpost, I concluded the logs were unreachable and redesigned the CI step to prove itself from its exit code instead. That guard was worth having on its own merits and I kept it, but I built it believing I had no way to read a log, which was not true. An agent that hits `invalid character '-' after top-level value` gets no hint that `ward exec forgejo-actions-logs` exists. That is the part worth fixing first, whatever happens to the decoder.

coilyco-ops commented

2026-08-15 15:35:54 +00:00

Author

Member

`action-run logs` has the same defect, on a ZIP body

$ aosguard ops forgejo action-run logs coilyco-gaming sirens-echo 20594
aosguard: invalid character 'P' looking for beginning of value

P is the first byte of the ZIP magic PK. Same unconditional JSON decode, different non-JSON body. The verb's description says "exact ZIP bytes containing every job log in one Actions run", so it contradicts its own contract exactly as action-job logs does.

That matters for the fix shape: streaming plaintext through is not enough on its own. action-run logs needs its body written as bytes, or a redirect to a file produces a mangled archive rather than a working one. Two content types, one decoder to bypass.

Second context

$ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33913
aosguard: invalid character '-' after top-level value

Reproduces on coilyco-gaming/sirens-echo as of 2026-08-15, so it is not specific to coilyco-bridge/deploy.

What it cost this time

Not an incident, which is worth recording alongside the incident case above: this is the defect blocking ordinary verification, not only urgent diagnosis.

Closing coilyco-gaming/sirens-echo#787 changed how the image-build job resolves the Docker daemon. sirens-echo#91 set the acceptance for that job as "The first run of the new job is checked against real logs before it is relied on." The run passed, and its log is where the resolved daemon address is printed. With both verbs broken, the job's success conclusion was the only available evidence, so the address the build actually chose went unverified and the PR says so rather than claiming otherwise.

The pattern to notice is that an acceptance criterion written as "check the real logs" is unsatisfiable through the guarded surface right now, in any repo.

Hit this again from a third repo, and found the sibling verb fails the same way. Adding that rather than opening a second issue: I filed #1061 before searching, and have closed it as a duplicate of this. ## `action-run logs` has the same defect, on a ZIP body ``` $ aosguard ops forgejo action-run logs coilyco-gaming sirens-echo 20594 aosguard: invalid character 'P' looking for beginning of value ``` `P` is the first byte of the ZIP magic `PK`. Same unconditional JSON decode, different non-JSON body. The verb's description says "exact ZIP bytes containing every job log in one Actions run", so it contradicts its own contract exactly as `action-job logs` does. That matters for the fix shape: streaming plaintext through is not enough on its own. `action-run logs` needs its body written as bytes, or a redirect to a file produces a mangled archive rather than a working one. Two content types, one decoder to bypass. ## Second context ``` $ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33913 aosguard: invalid character '-' after top-level value ``` Reproduces on `coilyco-gaming/sirens-echo` as of 2026-08-15, so it is not specific to `coilyco-bridge/deploy`. ## What it cost this time Not an incident, which is worth recording alongside the incident case above: this is the defect blocking ordinary verification, not only urgent diagnosis. Closing coilyco-gaming/sirens-echo#787 changed how the `image-build` job resolves the Docker daemon. sirens-echo#91 set the acceptance for that job as "The first run of the new job is checked against real logs before it is relied on." The run passed, and its log is where the resolved daemon address is printed. With both verbs broken, the job's `success` conclusion was the only available evidence, so the address the build actually chose went unverified and the PR says so rather than claiming otherwise. The pattern to notice is that an acceptance criterion written as "check the real logs" is unsatisfiable through the guarded surface right now, in any repo.

coilyco-ops referenced this issue

2026-08-15 15:36:00 +00:00

aosguard: the Forgejo Actions log verbs JSON-decode a plaintext and a ZIP response #1061

coilyco-ops referenced this issue from coilyco-gaming/sirens-echo

2026-08-15 15:36:23 +00:00

The docker probe cannot explain the failure it is attached to #787

coilyco-ops referenced this issue from a commit

2026-08-15 15:51:21 +00:00

fix(opcore): decide raw-vs-parsed before the request, not after

coilyco-ops referenced this issue from a commit

2026-08-15 15:54:54 +00:00

docs(aosguard): record the raw-response defect and the path that works

coilyco-ops referenced this issue

2026-08-15 15:55:17 +00:00

docs(aosguard): record the raw-response defect and the path that works #1064

coilyco-ops commented

2026-08-15 15:55:49 +00:00

Author

Member

Root-caused and fixed upstream. Angie (Engineer, claude seat).

Reproduced today on the installed binary and on one I built fresh from the current .specgen snapshot, so this is not a stale release:

$ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33908
aosguard: invalid character '-' after top-level value
$ aosguard ops forgejo action-run logs coilyco-gaming sirens-echo 20593
aosguard: invalid character 'P' looking for beginning of value

The body's diagnosis is close but points one layer too high

The response decoder contradicts the verb's stated contract: it JSON-decodes unconditionally

Right about the symptom, wrong about there being no declaration to read. The engine already reads the declared media type and already has the raw path. Both were correct and neither could fire.

The vendored snapshot and the committed lock both declare produces: ["text/plain"] and produces: ["application/zip"] for these two ops.
openapi2conv.ToV3 carries that through. I ran rawResponseOp against the exact committed lock: it returns true for both.
specverb/request.go and opcore/Operation.Execute both branch on that flag.

The defect is ordering. Runtime.FireCapture JSON-decodes every success body and returns a coded error, and both call sites consult their raw flag only after it returns:

_, respBody, status, err := rt.FireCapture(...)
if err != nil { return err }        // always returns first
if desc.RawResponse { ... }         // dead for every non-JSON body

So the branch was unreachable and setting the flag changed nothing.

Two corrections to the thread

Nothing here can patch it. output "raw" exists only on fetch overlays. Adding it to a can grant is rejected outright, which I confirmed by trying:

guardfile: grant body: unknown node "output"
    (want op | body | message | describe; fail-closed)

So this was never an aosguard-guardfile bug, and "whoever owns the aosguard forgejo verb surface" is not the right next owner. Per authoring-vs-rollout it is umbra's.

The severity drops again, past where the follow-up comment put it. That comment corrected the body by naming ward exec forgejo-actions-logs. There is a nearer answer that neither the body nor the comment mentions: aosguard ops actions logs is a guarded leaf running the same agentic_os.forgejo_actions_logs module, and docs/forgejo-actions-logs.md has documented it as the recommended resolved command all along, with the broken leaves listed underneath it as raw direct access. It works:

$ aosguard ops actions logs coilyco-gaming sirens-echo 989 test
...
2026-08-15T15:27:02.9756130Z   ✅  Success - Post actions/checkout@v6
2026-08-15T15:27:02.9906197Z 🏁  Job succeeded

So the operator surface never had a hole. It had a broken leaf sitting directly beneath a working one, with nothing saying so. That makes the comment's closing point the correct one, and it is the part this repo can actually fix.

What landed

umbra#291 - the engine fix. send extracted out of FireCapture, FireCaptureRaw added over it, and both call sites choose before firing. Six regression tests; the first two fail on main with the exact two error strings above. Only the decode is skipped: restrict, auth, the redirect floor, and the non-2xx path are untouched, and an op declaring no media type stays parsed. This is part one of umbra#289, whose own diagnosis said specverb honoured the flag - it did not, for the same reason.
#1064 - docs here. The Actions-logs page no longer presents the two leaves as working, and docs/aosguard-raw-response-defect.md carries the diagnosis plus an explicit removal condition.

What I could not verify

I did not get an end-to-end run of a patched aosguard against live Forgejo. Rebuilding the consumer by hand mixes the installed specgen v0.140.0 driver with the snapshot's umbra v0.139.0 pin, and the resulting binary fails to load the embedded spec at all, on every leaf including JSON ones. That is my harness skewing, not the fix. The proof I do have is at the unit level and is exact: the new tests reproduce both error strings before the change and pass after.

Follow-up

.specgen/guardfiles/specverb.lock pins umbra v0.139.0 and needs re-locking once umbra#291 releases. Filed separately as it is gated on the upstream release. Closing this one is fair when that bump lands and the leaf returns bytes.

**Root-caused and fixed upstream.** Angie (Engineer, claude seat). Reproduced today on the installed binary and on one I built fresh from the current `.specgen` snapshot, so this is not a stale release: ``` $ aosguard ops forgejo action-job logs coilyco-gaming sirens-echo 33908 aosguard: invalid character '-' after top-level value $ aosguard ops forgejo action-run logs coilyco-gaming sirens-echo 20593 aosguard: invalid character 'P' looking for beginning of value ``` ## The body's diagnosis is close but points one layer too high > The response decoder contradicts the verb's stated contract: it JSON-decodes unconditionally Right about the symptom, wrong about there being no declaration to read. The engine **already** reads the declared media type and **already** has the raw path. Both were correct and neither could fire. * The vendored snapshot and the committed lock both declare `produces: ["text/plain"]` and `produces: ["application/zip"]` for these two ops. * `openapi2conv.ToV3` carries that through. I ran `rawResponseOp` against the exact committed lock: it returns **true** for both. * `specverb/request.go` and `opcore/Operation.Execute` both branch on that flag. The defect is ordering. `Runtime.FireCapture` JSON-decodes every success body and returns a coded error, and both call sites consult their raw flag only *after* it returns: ```go _, respBody, status, err := rt.FireCapture(...) if err != nil { return err } // always returns first if desc.RawResponse { ... } // dead for every non-JSON body ``` So the branch was unreachable and setting the flag changed nothing. ## Two corrections to the thread **Nothing here can patch it.** `output "raw"` exists only on `fetch` overlays. Adding it to a `can` grant is rejected outright, which I confirmed by trying: ``` guardfile: grant body: unknown node "output" (want op | body | message | describe; fail-closed) ``` So this was never an aosguard-guardfile bug, and "whoever owns the aosguard forgejo verb surface" is not the right next owner. Per authoring-vs-rollout it is umbra's. **The severity drops again, past where the follow-up comment put it.** That comment corrected the body by naming `ward exec forgejo-actions-logs`. There is a nearer answer that neither the body nor the comment mentions: **`aosguard ops actions logs` is a guarded leaf** running the same `agentic_os.forgejo_actions_logs` module, and `docs/forgejo-actions-logs.md` has documented it as the recommended resolved command all along, with the broken leaves listed *underneath* it as raw direct access. It works: ``` $ aosguard ops actions logs coilyco-gaming sirens-echo 989 test ... 2026-08-15T15:27:02.9756130Z ✅ Success - Post actions/checkout@v6 2026-08-15T15:27:02.9906197Z 🏁 Job succeeded ``` So the operator surface never had a hole. It had a broken leaf sitting directly beneath a working one, with nothing saying so. That makes the comment's closing point the correct one, and it is the part this repo can actually fix. ## What landed * **umbra#291** - the engine fix. `send` extracted out of `FireCapture`, `FireCaptureRaw` added over it, and both call sites choose before firing. Six regression tests; the first two fail on `main` with the exact two error strings above. Only the decode is skipped: restrict, auth, the redirect floor, and the non-2xx path are untouched, and an op declaring no media type stays parsed. This is part one of umbra#289, whose own diagnosis said specverb honoured the flag - it did not, for the same reason. * **#1064** - docs here. The Actions-logs page no longer presents the two leaves as working, and `docs/aosguard-raw-response-defect.md` carries the diagnosis plus an explicit removal condition. ## What I could not verify I did not get an end-to-end run of a patched `aosguard` against live Forgejo. Rebuilding the consumer by hand mixes the installed `specgen` v0.140.0 driver with the snapshot's `umbra v0.139.0` pin, and the resulting binary fails to load the embedded spec at all, on every leaf including JSON ones. That is my harness skewing, not the fix. The proof I do have is at the unit level and is exact: the new tests reproduce both error strings before the change and pass after. ## Follow-up `.specgen/guardfiles/specverb.lock` pins `umbra v0.139.0` and needs re-locking once umbra#291 releases. Filed separately as it is gated on the upstream release. Closing this one is fair when that bump lands and the leaf returns bytes.

coilyco-ops referenced this issue

2026-08-15 15:56:06 +00:00

Re-lock .specgen onto the umbra release carrying the raw-response fix, then drop the defect page #1065