specgen build reuses a cached binary when a --umbra-replace checkout changes, so a dev iterating on umbra gets stale output silently #345

New issue

Open

opened 2026-08-30 05:26:14 +00:00 by coilyco-ops · 0 comments

coilyco-ops commented

2026-08-30 05:26:14 +00:00

Owner

Found while measuring #338, where it inverted a result and nearly ended the investigation on a wrong conclusion.

What happens

specgen lock --umbra-replace /path/to/umbra writes a dev lock pointing the generated module at a local checkout.
Edit umbra's source in that checkout.
specgen build --out ./bin reports built ./bin and exits 0.
The binary does not contain the edit. It is the previous build, copied.

Clearing ~/.umbra/cache and rebuilding produces a binary that does.

Measured

Same guardfile, same command, the only difference being a cleared cache:

stale cache build     p50 = 5195 ms
true rebuild          p50 =  383 ms

Those are the same source. The 5195ms binary predates a one-line change in pkg/mcpclient. Nothing said it was stale.

Cause

Per specgen-materialization.md, .stamp.json records input hashes for "member identities and bytes, spec contracts, dependency lock, and generator version". A rebuild fires only when one of those changes or the binary is missing.

A replace-target checkout's source is in none of them. The dep lock names the replace path, and that path does not change when the code behind it does. So the stamp is identical across an arbitrarily large source edit.

This is correct for the normal case: a released umbra pinned by version is immutable, so hashing it would be wasted work. It is only wrong for a replace, which is mutable by construction.

Why it is worth fixing rather than documenting

It fails in the most misleading direction available. A stale build reports success, and the operator's next act is to measure or test the thing they think they just changed. In #338 that produced a clean "hypothesis refuted" reading from a run that never contained the hypothesis.

It also only bites people working on umbra itself, which is a small group who will each lose the same hour.

Do

Options, cheapest first:

When the dep lock carries a replace to a local path, treat the cache as always stale and rebuild. Simple, correct, costs a go build on a path already understood to be a dev loop.
Hash the replace target's tracked .go files into the stamp. More precise, more code, and needs a decision about untracked files.
Refuse to use the cache at all when a dev lock is present, and say so on stderr.

Option 1 unless there is a reason a dev-lock rebuild is expensive enough to care about.

Acceptance

With a --umbra-replace lock, editing the replaced checkout and running specgen build produces a binary containing the edit, with no manual cache clearing.

Found while measuring #338, where it inverted a result and nearly ended the investigation on a wrong conclusion. ## What happens 1. `specgen lock --umbra-replace /path/to/umbra` writes a dev lock pointing the generated module at a local checkout. 2. Edit umbra's source in that checkout. 3. `specgen build --out ./bin` reports `built ./bin` and exits 0. 4. **The binary does not contain the edit.** It is the previous build, copied. Clearing `~/.umbra/cache` and rebuilding produces a binary that does. ## Measured Same guardfile, same command, the only difference being a cleared cache: ``` stale cache build p50 = 5195 ms true rebuild p50 = 383 ms ``` Those are the same source. The 5195ms binary predates a one-line change in `pkg/mcpclient`. Nothing said it was stale. ## Cause Per [specgen-materialization.md](https://forgejo.coilysiren.me/coilyco-flight-deck/umbra/src/branch/main/docs/specgen-materialization.md), `.stamp.json` records input hashes for "member identities and bytes, spec contracts, dependency lock, and generator version". A rebuild fires only when one of those changes or the binary is missing. A `replace`-target checkout's **source** is in none of them. The dep lock names the replace path, and that path does not change when the code behind it does. So the stamp is identical across an arbitrarily large source edit. This is correct for the normal case: a released umbra pinned by version is immutable, so hashing it would be wasted work. It is only wrong for a `replace`, which is mutable by construction. ## Why it is worth fixing rather than documenting It fails in the most misleading direction available. A stale build reports success, and the operator's next act is to measure or test the thing they think they just changed. In #338 that produced a clean "hypothesis refuted" reading from a run that never contained the hypothesis. It also only bites people working on umbra itself, which is a small group who will each lose the same hour. ## Do Options, cheapest first: 1. When the dep lock carries a `replace` to a local path, treat the cache as always stale and rebuild. Simple, correct, costs a `go build` on a path already understood to be a dev loop. 2. Hash the replace target's tracked `.go` files into the stamp. More precise, more code, and needs a decision about untracked files. 3. Refuse to use the cache at all when a dev lock is present, and say so on stderr. Option 1 unless there is a reason a dev-lock rebuild is expensive enough to care about. ## Acceptance With a `--umbra-replace` lock, editing the replaced checkout and running `specgen build` produces a binary containing the edit, with no manual cache clearing.

coilyco-ops referenced this issue

2026-08-30 05:26:30 +00:00

Bound the stdio close wait, cutting 5.2s off every call #344

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/umbra#345

No description provided.

Rows
Columns