[interactive] Rotate Telegram bot credential exposed in Alertmanager logs #708
Labels
No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
coilyco-flight-deck/infrastructure#708
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Alertmanager notification failures include the Telegram Bot API request URL in pod logs. That URL contains the bot credential. Do not copy the log line or credential into this issue.
Human wall:
Operator follow-through:
Escalated during the 2026-08-22 queue re-baseline (coilysiren/inbox#391). Kai confirmed today that this credential is still exposed and has not been rotated out of band.
This is now the oldest genuinely live P0 in the estate, and after the re-baseline it is one of only a handful of P0s whose urgency is real rather than inherited from a passed deadline.
Timeline as it stands:
For context on why this surfaced now: six P0s ranked above this one were tied to the August 19 Temporal Vibe Check demo and were closed today as delivered. This item outranked none of them by label, and was effectively buried under a deadline that had already passed. That ranking failure is the substance of #391.
Rotation is a live-system action owned by DevOps and is not this seat's to perform. No rotation was attempted here.
What the re-baseline recommends, in order:
Note that
coilysiren/inbox#289is an open P1 to roll out a shared Telegram CI failure action across the Forgejo fleet. If that rollout consumes the same credential, sequence it after rotation so it does not distribute a value that is about to be invalidated.Rotation landed. Consumer resync is blocked on runtime authority, detail below.
Kai revoked and re-minted at BotFather and wrote the replacement to SSM at
2026-08-22T21:20:34Z(parameter version 2). No credential value appears in this comment, in the session transcript, or in any command argv used to reach these findings.Verified
@coilyco_ops_bot(HTTP 200).Consumer audit
Five copies of this credential exist outside SSM. At audit time every one of them still held version 1.
fleet-reachability/fleet-reachability-telegramExternalSecret on kai-server. Last refresh20:32:22Z, 48 minutes before rotation. Has since self-healed on its 1h cycle at21:32:22Z.signoz-telegram/signoz-telegram-secretExternalSecret on kai-server. Last refresh20:33:58Z. Self-healed at21:33:58Z.~/.config/fleet-o11y/telegram-bot-tokenhost cache on ser8 and on kai-server. Both still byte-identical to version 1. These do not self-heal, because theo11y-telegramrole only rewrites them on an Ansible run.gatus(59d) andsignoz-telegram(21d).gatus(22d) andsignoz-telegram(2d12h).The ExternalSecret self-heal is not sufficient. Both Gatus and both mappers consume the token through
env.valueFrom.secretKeyRef, which Kubernetes resolves once at pod start and never updates in place. All four pods predate the rotation, so all four are running with the dead credential in their environment right now. Every one of them needs a rollout restart before delivery works again.Practical consequence worth stating plainly: ops-plane Telegram alerting has been failing 401 since 21:20:34Z and will keep failing silently until those restarts happen. Delivery failure is the failure mode this whole channel exists to prevent.
Root cause for the "stop emitting the token" item
The credential is shipped to the mapper pre-embedded in a URL. From the
signoz-telegram-secretExternalSecret target template:mcp-beaverreceives a base URL that is the credential, so any code path that reports a failing request by echoing its URL emits the token verbatim. That is the observed leak shape, and it explains why rotation alone cannot close this. Two separable fixes:TELEGRAM_BOT_TOKENand a credential-freeTELEGRAM_API_BASE_URL, and let the client assemble the path at call time.mcp-beavershould redact the bot path segment before a request URL reaches a log or an error string, so a future caller cannot reintroduce this by configuration. Acceptance condition: a deliberately failing Telegram call produces a log line containing/bot<redacted>/and no token material.Item 1 does not fully protect item 2's absence, and item 2 does not depend on item 1. Both should land.
Residual copies of the burned value
Blocked
Every mutating step was denied by this session's harness policy: the Ansible cache resync, the pod restarts, and the delivery drill. This is a runtime authority wall, not a boundary question. The exact actions still needed, in order:
just ansible-sync apply fleet tags=o11y-telegram hosts=ser8,kai-server(rewrites both host caches from SSM v2; the role never blanks a cache on failure).rollout restart deploy/gatus -n fleet-reachabilityandrollout restart deploy/signoz-telegram -n signoz-telegram, on both clusters.Sequencing note carried forward from the re-baseline:
coilysiren/inbox#289distributes this same credential to the Forgejo fleet and must run after step 1, not before.Correction to my previous comment, plus partial completion. kai-server is rotated and restarted. ser8 is still on the dead credential.
Correction
My previous comment said the fix was "restart the four pods on both clusters." That is wrong for ser8, and acting on it would have produced a false all-clear.
ser8 runs no External Secrets operator. Its Telegram Secrets are streamed in by an operator, not reconciled from SSM. From
services/signoz-telegram/README.mdincoilyco-bridge/deploy: ser8 "has no External Secrets operator, so an operator streams its Secrets over SSH." Confirmed against the live objects:signoz-telegram/signoz-telegram-secret-creationTimestamp: 2026-08-01T21:05:42Z, no owner reference.fleet-reachability/fleet-reachability-telegram-creationTimestamp: 2026-06-24T06:31:06Z.Neither has been touched since well before the rotation, so both still hold version 1. Restarting those two ser8 pods accomplished nothing except reloading the same dead value. I did restart them before establishing this, which was a wasted action on my part, though not a harmful one.
Done and verified
~/.config/fleet-o11y/telegram-bot-tokenon ser8 and kai-server both now match SSM version 2 byte-for-byte, confirmed by digest comparison.gatusandsignoz-telegramwere then restarted and came up at ~21:43Z, after those refreshes, so both pods now hold version 2.A bug found while applying the fix
just ansible-sync apply fleet tags=o11y-telegram hosts=ser8,kai-serverfails on ser8 withDestination directory /home/kai/.config/fleet-o11y does not exist.The
o11y-telegramrole resolves SSM underdelegate_to: localhostwithrun_once: true, then loops the copy overo11y_telegram_resolved.resultsusingitem.item.cache. Because the loop items were templated once, during therun_oncepass,o11y_telegram_cache_diris frozen to the first host'sHOME. kai-server iskaiand ser8 iscoilysiren, so every host after the first writes to the wrong path.This means the multi-host invocation has never actually refreshed ser8, and the failure is silent in the sense that an operator reading
changed=1on the recap could believe otherwise. Working around it by runninghosts=ser8alone succeeded, which is how ser8's cache got fixed above. The role should compose the cache path per-host rather than carrying it through the delegated loop item. Worth its own issue.Still outstanding
just rollout-signoz-telegram-ser8incoilyco-bridge/deploystreams the credential from SSM, redeploys, and sends one SigNoz channel test. That single command covers both the ser8 resync and the ser8 half of the delivery drill. It was denied by this session's harness policy, so it has not run.fleet-reachabilityrole (fleet_reachability_manage_telegram_secret), not by the rollout script. Needs its own play run, then a restart.Revision to the durable fix
My previous comment classified "stop templating the token into a URL" as an operational change in scope for this role. That was also wrong, and the manifest says so directly:
The
signoz-telegram.mcp.kdlguardfile takesbase-urlas a single env value.mcp-beaverhas anauth query-paramblock but no path-segment composition, and Telegram puts the credential in the path rather than a header or query parameter. So the chart cannot be fixed from the deploy side alone. This needs one of:mcp-beavergains path composition, so the token is supplied as its own env var and assembled at call time, ormcp-beaverredacts the bot path segment before any request URL reaches a log line or error string.Both are source changes in
mcp-beaverand belong to Engineering. The second is the stronger fix, because it holds even if a future caller reintroduces a credential-bearing base URL by configuration.That leaves this role's own fallback, which is the one the follow-through list already anticipated: document the upstream limitation and restrict retention and access on the logs that carry it. That part is mine and is not yet done.
Rotation and resync complete across both clusters. One confirmation is Kai's, and the durable fix is still open.
Final state
Credential. SSM version 2 authenticates as
@coilyco_ops_bot. Version 1 returns 401. No credential value crossed a log, a transcript, or argv at any point in this work.Host caches. ser8 and kai-server both match version 2, verified by digest.
kai-server cluster. Both ExternalSecrets reconciled to version 2 (21:32:22Z, 21:33:58Z).
gatusandsignoz-telegramrestarted afterward at ~21:43Z, so both pods hold version 2.ser8 cluster. Neither Secret self-heals here, so both were re-streamed from SSM by hand:
fleet-reachability-telegramre-streamed byplaybooks/fleet-reachability.yml(changed=1). That play does not restart the Deployment when the manifest is unchanged, sogatusneeded a separate rollout restart afterward. Worth knowing: updating that Secret alone changes nothing until the pod is restarted, and the play will not do it for you.signoz-telegram-secretre-streamed byjust rollout-signoz-telegram-ser8, which also reconciled the SigNoz webhook channel and sent one channel test. Result:SigNoz accepted the Telegram channel test.What is proven and what is not
The ser8 channel test is a real end-to-end call: SigNoz to the mapper to the Telegram API on the rotated credential. That is the delivery drill this issue asked for, and it passed.
kai-server has not been independently drilled. Its mapper is exercised by Forgejo CI failures rather than by a channel test, and the rollout path for that instance sends no test. What is established there is narrower: the Secret provably holds version 2, and the pod provably started after the reconcile. The credential is correct. Delivery through that specific pod is inferred from the same credential working from ser8, not observed.
The one confirmation I cannot make: whether the test message actually landed in the red channel. The mapper returning success means Telegram accepted the request. Kai seeing it on a phone is the only real acceptance evidence, and until that happens this is a strong signal rather than proof.
Deliberately not done
I did not run a failure-path drill. The leak only reproduces on a failed notification, so triggering one would write the freshly rotated token straight back into the logs this issue exists to clean up. Reproducing the bug and re-burning the credential is not a test worth running.
The mapper's success path logs nothing at all. Scanned the ser8 pod's output for any
/bot<id>:<secret>URL path and found none, but that is evidence about the quiet path only, and says nothing about the failure path.Landed
infrastructure#900merged. Theo11y-telegramrole composed its cache path from adelegate_to: localhost+run_once: trueloop item, freezing it to the first host'sHOME, so any multi-host run wrote ser8's files into/home/kaiand failed. ser8's cache had therefore never been refreshed by a multi-host invocation. Params now carry basenames, the copy task composes per host, andhosts=ser8,kai-serversucceeds on both and is idempotent.Still open
mcp-beaverneeds either path composition, so the token stops being half of a base URL, or redaction of the bot path segment before any URL reaches a log or error string. Redaction is the stronger of the two, because it survives a future caller reintroducing a credential-bearing base URL by configuration. Both are source changes and belong to Engineering. Acceptance condition: a deliberately failed Telegram call logs/bot<redacted>/and no token material.coilysiren/inbox#289may now proceed. It distributes this credential to the Forgejo fleet and had to wait for the rotation, which is done.This issue should not close until item 1 lands. Rotation without it means the next notification failure burns the new credential exactly as it burned the old one.
Retention question answered, and the emitting unit is not the one this issue names.
Kai confirmed the drill message arrived in the red channel, so delivery on the rotated credential is now observed rather than inferred. The channel is fully recovered.
The exposed value did reach log retention
Queried SigNoz with count-only aggregations, deliberately never fetching a log body, so no credential entered the session.
body CONTAINS 'api.telegram.org/bot', last 30 days: 36 records.That closes the re-baseline's item 2. The answer is yes, it reached retention, and the 36 retained records carry the now-dead version 1.
The emitter is Gatus, not Alertmanager and not the mapper
Grouped by namespace and container, all 36 records resolve to a single source:
Nothing from
observability, nothing fromsignoz-telegram. This matters because it contradicts both this issue's framing and my own earlier analysis:mcp-beaver#93against the mapper. The mapper genuinely does hold the token as half of a base URL, so that issue is still worth doing as hardening, but it is not the unit that caused this exposure and fixing it alone would have left the real leak wide open. I will correct that issue rather than leave it claiming to be the root cause.Gatus keeps its own direct Telegram path by design, deliberately independent of the mapper so it survives the cluster it watches. It takes
TELEGRAM_BOT_TOKENas env and builds the Bot API URL itself, and on a failed send it logs that URL.This is recurring, not historical
Daily buckets put the 36 records in two clusters: roughly 12 around 2026-08-02 and 24 around 2026-08-20.
The most recent burst is about two days ago, three weeks after this issue was opened and while it sat unactioned. This is not a one-time historical spill. It fires whenever a Gatus Telegram send fails, and a Gatus Telegram send failing is precisely the condition this channel exists to report. The next occurrence writes the new credential into retention exactly as the last one wrote the old.
What the fix actually is
ghcr.io/twin/gatusis third-party, so patching the emitter is not available. The issue's own follow-through anticipated this case: document the upstream limitation and restrict retention and access. Concretely:signoz-pipelinesprocessor that masks the/bot<id>:<secret>path segment in log bodies. Version-controlled log ingest parsers are exactly the right layer, this is deploy-owned operational config, and it protects against any future unit that logs a credential-bearing Telegram URL rather than just this one.mcp-beaver#93as hardening for the mapper's URL-as-credential shape, correctly labelled as defence in depth rather than the cause.Item 1 is the one that matters and it should not wait. The credential is live, the trigger is common, and every day this stays open is another chance to burn a token that was rotated an hour ago.
One stated uncertainty: I did not fetch any log body, so I have not visually confirmed the token appears inside those 36 records. The evidence is the container identity plus the
api.telegram.org/botmatch plus this issue's original first-hand report. That is strong but it is inference, and confirming it would mean handling the value.Measured 2026-08-22. The credential is real and unrotated, but this issue names the wrong component and mischaracterises the pattern. No token value is reproduced below - counts and locations only.
The emitter is Gatus, not Alertmanager
Every occurrence in the last 30 days comes from one pod:
Nothing from Alertmanager, and nothing from any other workload. The title's "exposed in Alertmanager logs" has sent anyone investigating to the wrong place for 22 days. The unit fix belongs in the Gatus configuration, where a Telegram alerting provider is presumably configured with the bot token embedded in the endpoint URL, and the URL is logged on failure. That last clause is inference from the shape of the finding, not something verified in the config, and whoever picks this up should confirm it before editing.
The pattern is episodic, not a continuous drip
Counted with
body CONTAINS 'api.telegram.org/bot', daily buckets over 30 days:So it is not leaking hourly. It fires in bursts, which is consistent with a health-check alert path that only logs the endpoint when a probe fails. 36 records carrying the credential now sit in SigNoz log retention and are readable by anyone with SigNoz access, plus wherever that ClickHouse data is backed up.
It is not dormant - it recurred yesterday
The most important correction. This issue was filed 07-31 and last touched 08-05, and the quiet period since could easily read as "the problem went away." It did not. The largest burst was 2026-08-21, twenty-one days after filing. Any assessment that treats this as a stale one-time leak is wrong.
A confounder that must not be mistaken for a fix
The emitting pod
gatus-7f4898c65f-5x8s4no longer exists. Gatus is now running asgatus-59c8d844f7-f2hq8, created 2026-08-22T21:58:30Z - a different ReplicaSet hash, so this was a redeploy rather than a restart.signoz-telegramwas also recreated at 21:58:57Z, within thirty seconds, which looks like one coordinated rollout.Do not read the current zero as evidence of a fix. Emission had already stopped more than 24 hours before that rollout, and the gap between the 08-02 and 08-21 bursts was nineteen days. A quiet window here proves nothing. The only thing that would demonstrate a fix is a probe failure occurring after the rollout with no credential in the resulting log line.
Recommended sequence, unchanged in substance
coilysiren/inbox#289- the shared Telegram CI failure action across the Forgejo fleet - after rotation, or it will distribute a value that is about to be invalidated.Rotation and config changes are DevOps actions and were not performed here. This comment is observation only.
Method note
signoz_aggregate_logssilently ignores itssearchTextparameter - a nonsense control string returned the same 7,967,817 rows as the real query, which is simply the unfiltered total. Thefilterparameter withbody CONTAINSworks correctly and returns 0 for the same control. Every number above usesfilter. Worth knowing before anyone else counts anything in these logs, and it is the same silent-filter-drop failure class recorded against Forgejo label queries incoilysiren/inbox#391.Rotation done, 2026-08-22. Kai rotated the Telegram bot credential roughly ten minutes after the measurement above was posted, closing step 1 of the recommended sequence. The 22-day exposure window is ended.
Ops has picked this up and flagged that the exposure path itself is not yet closed - which matches the measurement: rotation invalidates the leaked value but does nothing about the Gatus alerting config that emits it. The next probe failure would put a fresh credential into the same log stream. Steps 2 through 4 remain open and are with ops:
coilysiren/inbox#289is no longer blocked by rotation ordering, since the rotation has happened. It is now only gated on step 2, so that the fleet-wide Telegram action is not rolled out onto a path that still leaks.The component correction in the previous comment stands and is the load-bearing part for whoever does step 2: the emitter is Gatus in
fleet-reachability, not Alertmanager, despite this issue's title.Recorded by the exec seat. No rotation, config change, or log purge was performed here.
Durable fix landed and verified. Every follow-through item on this issue is now done.
The leak is closed at ingest
coilyco-bridge/deploy#776merged and applied to both clusters.Two separate gaps, both real, and fixing either alone would have left the leak open:
authorization|password|secret|token|api[_-]?keyand similar. A Telegram bot URL contains none of those words, because the token is the path segment. Copying kai-server's processors to ser8 unchanged would have produced a config that looked correct and redacted nothing.Both clusters now run a credential-in-URL rule ahead of the keyword rule. It masks the token segment rather than blanking the body, because the emitting line is an alert-delivery failure and discarding it would trade a credential leak for a blind spot in the channel that reports outages.
Verified end to end, on the cluster that leaked
Emitted a deliberately fake bot-token-shaped URL from a throwaway pod on ser8 and read back what SigNoz stored:
The token segment is gone and the host, endpoint, and error reason survive. This is the actual acceptance path rather than a rendered config or a healthy pod.
Also confirmed the change did not cost anything: ser8 kept ingesting through the rollout (627 records in the following 5 minutes, cluster attribution intact), and both collector rollouts reported healthy.
Follow-through checklist
This issue is ready to close.
What remains, tracked elsewhere
coilyco-flight-deck/mcp-beaver#93- hardening for the mapper's base-url-is-a-credential shape. It never leaked, but a caller adding request-URL logging would turn it into this incident. Not a blocker here.infrastructure#900merged separately: theo11y-telegramrole froze its cache path to the first host'sHOME, so ser8's cache had never once been refreshed by a multi-host run.Operator note
This host had no
ser8kube context, thoughSER8_KUBE_CONTEXTinrollout-ser8-host-metrics.shassumes an operator has one. Merged ser8's ownser8entry into the local kubeconfig with Kai's approval, leavingcurrent-contexton kai-server. Prior kubeconfig backed up at~/.kube/config.bak-708. The ser8 release went 4 -> 5, so rollback ishelm --kube-context ser8 -n observability rollback k8s-infra 4.The mechanism blocker cleared.
mcp-beaver#93landed as31eb74conmain.This issue's own note said it should not close until that landed, because rotation replaces the burned value and does nothing about the mechanism, so the next notification failure burns the new credential the same way.
What changed
mcp-beaver now resolves its base URL once at startup and registers that URL's path as unemittable. The registered prefix is masked in the structured log line and in the error string handed back to the caller, which were the same string with only the first one guarded.
The whole operator-supplied prefix is masked rather than a
bot<digits>:<secret>pattern, because a pattern would hold only until an upstream formatted its credential differently, and would then fail silently.Details in
docs/logs.md.What this issue still needs before closing
A rolled image. The fix is on
mcp-beavermain and thesignoz-telegrammapper is running whatever tag its manifest pins. Until that pod carries the new binary, the mechanism is unchanged in production.That is a deploy step on a hosted surface, so it is not mine to take. Handing it over with the exact action: roll
services/signoz-telegramincoilyco-bridge/deployonto anmcp-beaverimage built from31eb74cor later, then deliberately fail one notification and confirm the log line and the tool error both carry/<redacted>and no token material.Filed by Angie (platform seat) while working
coilysiren/inbox#426.