coilyco-flight-deck/agentic-os

Watch

trufflehog catalog hook scans gitignored target/ in Rust consumers, blocks every commit #288

New issue

Closed

opened 2026-06-25 11:23:05 +00:00 by coilysiren · 5 comments

coilysiren commented

2026-06-25 11:23:05 +00:00

Owner

Problem

The catalog trufflehog (secret scan, offline) hook shipped from this repo
(.pre-commit-hooks.yaml) false-positives on Rust build artifacts. On any host
that has run a release build, target/ exists (gitignored, untracked) and the
hook reads its binary blobs (libzerocopy-*.rlib, .rmeta, *.pdb), emitting
thousands of bogus "unverified" hits that block every git commit.

Reported in coilyco-bridge/agentic-os-kai#530. It blocked landing the .forgejo/
migration in coilysiren/repo-recall on a tower until rm -rf target/ (4.6GB,
fully reproducible). It affects every Rust-shaped consumer of this hook
(repo-recall, cli-guard, ward, coily).

Current hook entry

.pre-commit-hooks.yaml, id trufflehog:

entry: trufflehog git file://. --since-commit HEAD --no-verification --no-update --fail

Root cause

trufflehog's git file://. scan reads the working copy, and for untracked files
it does not consult .gitignore (upstream feature request trufflesecurity/trufflehog#3356
confirms trufflehog does not honor .gitignore). So gitignored target/ gets
walked. Removing target/ is what clears the hits, which proves the working-tree
read is the source.

Fix shape (verify empirically with a live trufflehog before landing)

This repo is the right home because the entry is centrally managed: consumers
carry it via the apply-agentic-os-hooks.py managed block, and that block is
regenerated with no per-hook args, so a consumer-side override does not survive.

Candidates, in rough order of preference:

--exclude-globs appended inline (travels with the shared entry, no per-repo
file needed): e.g. --exclude-globs 'target/**,dist/**,build/**,node_modules/**'.
Caveat to verify: --exclude-globs is documented as filtering at the git log
level, so confirm it also suppresses the untracked working-tree read, not just
committed objects.
--exclude-paths <file> of regexes (the pattern already used in
agentic-os-kai's .github/workflows/trufflehog.yml CI scan: (^|/)target/,
\.venv/, node_modules/, the cache dirs). Downside: needs a committed
regex file each consumer carries, awkward for a shared entry.
Scope strictly to the staged diff so untracked files are never read.

The carrying agent should reproduce with cargo build --release + an empty
commit, pick whichever flag actually suppresses the working-tree read, and keep
real staged-secret detection intact (do not blanket --only-verified, the hook
is offline --no-verification).

Rollout (part of done)

Landing the entry change is not enough on its own. Consumers pin a rev:, so
after this lands and a release tag cuts, bump the pin fleet-wide via
make apply-agentic-os-hooks / apply-agentic-os-hooks.py --rev vX so Rust
repos actually receive the fix. Closing agentic-os-kai#530 depends on that
rollout reaching the affected repos.

Repro

cd <a Rust repo with this hook>
cargo build --release
git commit --allow-empty -m "trigger"   # blocks on trufflehog target/ hits

Filed from the agentic-os-kai#530 carry container, which is scoped to
agentic-os-kai and cannot push this repo. Cross-ref: coilyco-bridge/agentic-os-kai#530.

## Problem The catalog `trufflehog (secret scan, offline)` hook shipped from this repo (`.pre-commit-hooks.yaml`) false-positives on Rust build artifacts. On any host that has run a release build, `target/` exists (gitignored, untracked) and the hook reads its binary blobs (`libzerocopy-*.rlib`, `.rmeta`, `*.pdb`), emitting thousands of bogus "unverified" hits that block every `git commit`. Reported in coilyco-bridge/agentic-os-kai#530. It blocked landing the `.forgejo/` migration in `coilysiren/repo-recall` on a tower until `rm -rf target/` (4.6GB, fully reproducible). It affects every Rust-shaped consumer of this hook (repo-recall, cli-guard, ward, coily). ## Current hook entry `.pre-commit-hooks.yaml`, id `trufflehog`: entry: trufflehog git file://. --since-commit HEAD --no-verification --no-update --fail ## Root cause trufflehog's `git file://.` scan reads the working copy, and for untracked files it does not consult `.gitignore` (upstream feature request trufflesecurity/trufflehog#3356 confirms trufflehog does not honor `.gitignore`). So gitignored `target/` gets walked. Removing `target/` is what clears the hits, which proves the working-tree read is the source. ## Fix shape (verify empirically with a live trufflehog before landing) This repo is the right home because the entry is centrally managed: consumers carry it via the `apply-agentic-os-hooks.py` managed block, and that block is regenerated with no per-hook args, so a consumer-side override does not survive. Candidates, in rough order of preference: - `--exclude-globs` appended inline (travels with the shared entry, no per-repo file needed): e.g. `--exclude-globs 'target/**,dist/**,build/**,node_modules/**'`. Caveat to verify: `--exclude-globs` is documented as filtering at the `git log` level, so confirm it also suppresses the untracked working-tree read, not just committed objects. - `--exclude-paths <file>` of regexes (the pattern already used in agentic-os-kai's `.github/workflows/trufflehog.yml` CI scan: `(^|/)target/`, `\.venv/`, `node_modules/`, the cache dirs). Downside: needs a committed regex file each consumer carries, awkward for a shared entry. - Scope strictly to the staged diff so untracked files are never read. The carrying agent should reproduce with `cargo build --release` + an empty commit, pick whichever flag actually suppresses the working-tree read, and keep real staged-secret detection intact (do not blanket `--only-verified`, the hook is offline `--no-verification`). ## Rollout (part of done) Landing the entry change is not enough on its own. Consumers pin a `rev:`, so after this lands and a release tag cuts, bump the pin fleet-wide via `make apply-agentic-os-hooks` / `apply-agentic-os-hooks.py --rev vX` so Rust repos actually receive the fix. Closing agentic-os-kai#530 depends on that rollout reaching the affected repos. ## Repro cd <a Rust repo with this hook> cargo build --release git commit --allow-empty -m "trigger" # blocks on trufflehog target/ hits Filed from the agentic-os-kai#530 carry container, which is scoped to agentic-os-kai and cannot push this repo. Cross-ref: coilyco-bridge/agentic-os-kai#530.

coilysiren referenced this issue

2026-06-25 11:23:20 +00:00

trufflehog catalog hook scans gitignored target/ in Rust consumers, blocks every commit #287

coilysiren commented

2026-06-25 22:27:50 +00:00

Author

Owner

Carry-readiness notes (from the agentic-os-kai#530 re-dispatch)

Re-confirmed this from a kai-scoped carry container and pinned down the open empirical question as far as docs allow, so whoever carries this lands a near-mechanical diff.

Why the working-tree read happens (mechanism). trufflehog's git file://. source scans the working-tree state (staged + unstaged + untracked) as a synthetic diff against HEAD - that is precisely why it works as a pre-commit hook, catching secrets before they are committed. It does not honor .gitignore for those untracked files (upstream trufflesecurity/trufflehog#3356). So gitignored-but-untracked target/ blobs get walked. --since-commit HEAD does not change this - the working-tree synthetic diff is independent of the commit range.

Generator confirmed to wipe consumer-side args. apply-agentic-os-hooks.py:180 stamps bare - id: {h} lines with no args, so any args: an affected repo adds to its managed block is erased on the next rollout. Confirms this repo is the only durable home.

Flag recommendation, in order:

--exclude-paths=<file> is the documented path-regex filter for the git source (one regex per line, matched against object paths). It is the same approach already proven in agentic-os-kai's own .github/workflows/trufflehog.yml. Proven regex set to reuse: (^|/)target/, (^|/)\.venv/, (^|/)venv/, (^|/)node_modules/, (^|/)__pycache__/, (^|/)\.mypy_cache/, (^|/)\.pytest_cache/, (^|/)\.ruff_cache/ - add (^|/)(dist|build)/ for Rust/JS artifacts. Downside unchanged: needs a committed regex file in every consumer, awkward for a single shared entry.
--exclude-globs 'target/**,dist/**,build/**,node_modules/**' is inline (no per-consumer file) and is the cleaner shape if it covers the untracked working-tree read. That coverage is undocumented for the git source - docs describe --exclude-globs at the git log level. This is the one thing that still needs a live trufflehog against a real target/ to confirm before landing.

Carry environment requirement. The verify step needs the trufflehog binary plus a populated target/ (cargo build --release in a Rust consumer, then an empty commit). The kai-scoped container has neither (Python repo, no binary), which is why this cannot be verified from there. Carry this in a container that grants this repo and has trufflehog on PATH.

Rollout is part of done: land entry change -> release tag cuts -> bump rev: fleet-wide via apply-agentic-os-hooks.py --rev vX. agentic-os-kai#530 stays open until the rollout reaches the Rust consumers (repo-recall, cli-guard, ward, coily).

Cross-ref: coilyco-bridge/agentic-os-kai#530.

## Carry-readiness notes (from the agentic-os-kai#530 re-dispatch) Re-confirmed this from a kai-scoped carry container and pinned down the open empirical question as far as docs allow, so whoever carries this lands a near-mechanical diff. **Why the working-tree read happens (mechanism).** trufflehog's `git file://.` source scans the working-tree state (staged + unstaged + untracked) as a synthetic diff against HEAD - that is precisely why it works as a pre-commit hook, catching secrets before they are committed. It does **not** honor `.gitignore` for those untracked files (upstream trufflesecurity/trufflehog#3356). So gitignored-but-untracked `target/` blobs get walked. `--since-commit HEAD` does not change this - the working-tree synthetic diff is independent of the commit range. **Generator confirmed to wipe consumer-side args.** `apply-agentic-os-hooks.py:180` stamps bare ` - id: {h}` lines with no args, so any `args:` an affected repo adds to its managed block is erased on the next rollout. Confirms this repo is the only durable home. **Flag recommendation, in order:** - `--exclude-paths=<file>` is the documented path-regex filter for the **git** source (one regex per line, matched against object paths). It is the same approach already proven in agentic-os-kai's own `.github/workflows/trufflehog.yml`. Proven regex set to reuse: `(^|/)target/`, `(^|/)\.venv/`, `(^|/)venv/`, `(^|/)node_modules/`, `(^|/)__pycache__/`, `(^|/)\.mypy_cache/`, `(^|/)\.pytest_cache/`, `(^|/)\.ruff_cache/` - add `(^|/)(dist|build)/` for Rust/JS artifacts. Downside unchanged: needs a committed regex file in every consumer, awkward for a single shared entry. - `--exclude-globs 'target/**,dist/**,build/**,node_modules/**'` is inline (no per-consumer file) and is the cleaner shape **if it covers the untracked working-tree read**. That coverage is undocumented for the git source - docs describe `--exclude-globs` at the `git log` level. **This is the one thing that still needs a live trufflehog against a real `target/` to confirm** before landing. **Carry environment requirement.** The verify step needs the trufflehog binary plus a populated `target/` (`cargo build --release` in a Rust consumer, then an empty commit). The kai-scoped container has neither (Python repo, no binary), which is why this cannot be verified from there. Carry this in a container that grants this repo and has trufflehog on PATH. **Rollout is part of done:** land entry change -> release tag cuts -> bump `rev:` fleet-wide via `apply-agentic-os-hooks.py --rev vX`. agentic-os-kai#530 stays open until the rollout reaches the Rust consumers (repo-recall, cli-guard, ward, coily). Cross-ref: coilyco-bridge/agentic-os-kai#530.

coilysiren added the

priority

autonomy

headless

labels

2026-07-01 04:29:18 +00:00

coilyco-ops commented

2026-07-01 04:36:46 +00:00

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-agentic-os-288 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T04:36:43Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agentic-os-288` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T04:36:43Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-01 04:48:33 +00:00

fix(trufflehog): exclude gitignored build/cache dirs from secret scan

coilysiren closed this issue

2026-07-01 04:48:33 +00:00

coilysiren commented

2026-07-01 04:49:01 +00:00

Author

Owner

WARD-OUTCOME: done - trufflehog catalog hook now excludes gitignored build/cache dirs via scripts/trufflehog-scan.sh (--exclude-paths), landed on main d4ef2bb.

The interesting part was that the bug did not reproduce against the shipped trufflehog (3.95.7): its git file://. source only scans committed history plus the staged index, and never touches the untracked gitignored target/. So the reported flood must come from an older trufflehog whose synthetic working-tree diff included untracked files. That reframed the job from "reproduce and fix" to "make the shared entry robust across the version that does the untracked read."

What fought back was the flag choice. The issue's preferred candidate, inline --exclude-globs, turned out to be the wrong tool: I staged a secret inside target/ and --exclude-globs 'target/**' did NOT drop it - it only filters committed git-log objects, not the synthetic working/staged diff. --exclude-paths (a regex file) DID filter that diff by path while keeping a real leak outside the excluded dirs failing. That empirical split is the whole reason for the shape that landed.

To dodge the "every consumer carries an exclude file" downside I moved the entry to language: script pointing at a wrapper committed here, so the regex list ships centrally through the pre-commit cache. Verified end to end with pre-commit try-repo: gitignored target/ full of secrets passes, a real staged secret fails. Confidence is high on correctness.

One rough edge worth flagging: the fix is version-robust but I could not exercise the actual untracked-read path, since no old-enough trufflehog was on hand - I proved --exclude-paths filters the same synthetic-diff mechanism the old read rode, which is strong but indirect. Follow-up, and the real closer for agentic-os-kai#530: after this release tag cuts, the fleet rev: pin still needs bumping on the Rust consumers (repo-recall, cli-guard, ward, coily). That rollout is a host/ansible step outside this container's grant, so #530 should stay open until it lands there.

WARD-OUTCOME: done - trufflehog catalog hook now excludes gitignored build/cache dirs via scripts/trufflehog-scan.sh (--exclude-paths), landed on main d4ef2bb. The interesting part was that the bug did not reproduce against the shipped trufflehog (3.95.7): its `git file://.` source only scans committed history plus the staged index, and never touches the untracked gitignored `target/`. So the reported flood must come from an older trufflehog whose synthetic working-tree diff included untracked files. That reframed the job from "reproduce and fix" to "make the shared entry robust across the version that does the untracked read." What fought back was the flag choice. The issue's preferred candidate, inline `--exclude-globs`, turned out to be the wrong tool: I staged a secret inside `target/` and `--exclude-globs 'target/**'` did NOT drop it - it only filters committed git-log objects, not the synthetic working/staged diff. `--exclude-paths` (a regex file) DID filter that diff by path while keeping a real leak outside the excluded dirs failing. That empirical split is the whole reason for the shape that landed. To dodge the "every consumer carries an exclude file" downside I moved the entry to `language: script` pointing at a wrapper committed here, so the regex list ships centrally through the pre-commit cache. Verified end to end with `pre-commit try-repo`: gitignored `target/` full of secrets passes, a real staged secret fails. Confidence is high on correctness. One rough edge worth flagging: the fix is version-robust but I could not exercise the actual untracked-read path, since no old-enough trufflehog was on hand - I proved `--exclude-paths` filters the same synthetic-diff mechanism the old read rode, which is strong but indirect. Follow-up, and the real closer for agentic-os-kai#530: after this release tag cuts, the fleet `rev:` pin still needs bumping on the Rust consumers (repo-recall, cli-guard, ward, coily). That rollout is a host/ansible step outside this container's grant, so #530 should stay open until it lands there.

coilyco-ops commented

2026-07-01 07:04:29 +00:00

Owner

🔒 Reserved by ward agent --driver claude — container engineer-claude-agentic-os-288 on host kais-macbook-pro-2.local is carrying this issue (reserved 2026-07-01T07:04:27Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-agentic-os-288` on host `kais-macbook-pro-2.local` is carrying this issue (reserved 2026-07-01T07:04:27Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides.  — Claude (she/her), via `ward agent`

coilysiren commented

2026-07-01 07:07:43 +00:00

Author

Owner

WARD-OUTCOME: done - trufflehog wrapper (scripts/trufflehog-scan.sh, --exclude-paths) already landed on main d4ef2bb closing #288; I independently re-verified it end to end.

Candid note: I came up to a fully-carried issue. A prior container (linux-0c646...cc47) had already implemented the wrapper, pushed d4ef2bb straight to canonical main, closed the issue, and left its own done retrospective. My HEAD was already sitting exactly on that commit, so there was no feature left to build - the honest job was to re-verify rather than re-do.

What I checked, and where it fought back a little: my first "real staged secret still fails" test came back green (exit 0) and briefly worried me, until I realized I'd used AKIA...EXAMPLE, which is trufflehog's own blessed placeholder that detectors deliberately ignore. Swapping in a genuinely detectable token flipped it to a proper failure (exit 183), while the same token confined to a gitignored target/ passed clean. So the two load-bearing behaviors both hold: real staged leaks still block, gitignored build dirs don't. shellcheck passes on the wrapper and the full suite is green at HEAD.

Confidence: high on correctness of what landed. The one honest caveat carries over from the original carry - trufflehog 3.95.7 doesn't walk untracked target/ anyway, so the exclude is proven against the synthetic-diff mechanism but not against the actual old-version untracked read that triggered the report. Follow-up unchanged and real: this only reaches the Rust consumers after a release tag cuts and the fleet rev: pin is bumped (apply-agentic-os-hooks.py --rev vX), which is a host/ansible step outside this container. agentic-os-kai#530 should stay open until that rollout lands.

WARD-OUTCOME: done - trufflehog wrapper (scripts/trufflehog-scan.sh, --exclude-paths) already landed on main d4ef2bb closing #288; I independently re-verified it end to end. Candid note: I came up to a fully-carried issue. A prior container (linux-0c646...cc47) had already implemented the wrapper, pushed d4ef2bb straight to canonical main, closed the issue, and left its own done retrospective. My HEAD was already sitting exactly on that commit, so there was no feature left to build - the honest job was to re-verify rather than re-do. What I checked, and where it fought back a little: my first "real staged secret still fails" test came back green (exit 0) and briefly worried me, until I realized I'd used `AKIA...EXAMPLE`, which is trufflehog's own blessed placeholder that detectors deliberately ignore. Swapping in a genuinely detectable token flipped it to a proper failure (exit 183), while the same token confined to a gitignored `target/` passed clean. So the two load-bearing behaviors both hold: real staged leaks still block, gitignored build dirs don't. shellcheck passes on the wrapper and the full suite is green at HEAD. Confidence: high on correctness of what landed. The one honest caveat carries over from the original carry - trufflehog 3.95.7 doesn't walk untracked `target/` anyway, so the exclude is proven against the synthetic-diff mechanism but not against the actual old-version untracked read that triggered the report. Follow-up unchanged and real: this only reaches the Rust consumers after a release tag cuts and the fleet `rev:` pin is bumped (apply-agentic-os-hooks.py --rev vX), which is a host/ansible step outside this container. agentic-os-kai#530 should stay open until that rollout lands.

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

coilyco-flight-deck/agentic-os#288

No description provided.

Rows
Columns