coilyco-gaming/eco-app
Watch
2
0
Code Issues 33 Pull requests Projects Releases Packages 1 Wiki Activity Actions

pollution numbers haven't been updated in >12 hours #184

New issue
Open
opened 2026-07-29 21:26:09 +00:00 by coilysiren · 3 comments
coilysiren commented 2026-07-29 21:26:09 +00:00
Owner
Copy link

Incident

The public Eco World page was observed showing pollution numbers that had not changed for more than 12 hours on 2026-07-29.

The issue is not yet classified. The stale value may come from the Eco source observation, backend fetch or cache behavior, browser page-load behavior, or incorrect timestamp presentation. Do not assume the data plane or fix before collecting evidence.

Outcome

Identify the layer that stopped advancing, restore truthful current behavior where a defect exists, and prevent the page from implying that stale pollution data is current.

Freshness presentation work belongs in #183. This issue owns the current stale-data diagnosis and any smallest necessary repair.

Required evidence

  • Record the pollution value and every available source-observation, backend-fetch, cache, and browser-load timestamp without recording private player data.
  • Compare the public page with the corresponding read-only API or MCP response.
  • Determine whether a normal page reload changes the value or timestamp.
  • Classify the failure as source data, backend ingestion, cache behavior, browser behavior, presentation, or an external live-system dependency.
  • If the defect is repository-local, prove one fix locally before pushing.
  • If diagnosis or verification requires a live mutation, stop at an interactive operator handoff with the exact action and expected evidence.
  • Do not probe CI or production through repeated speculative pushes.

Acceptance criteria

  • The issue records which layer was stale and the evidence supporting that classification.
  • A repository defect has focused tests covering the stale path.
  • A live-only defect has an explicit operator action and verification result.
  • The page either advances on its truthful cadence or visibly reports that the pollution data is stale or unknown.
  • The result does not borrow the world snapshot timestamp for a different data plane.
  • Current validation passes, and any production verification names the deployed image and observed timestamp behavior.

Relationship to the campaign

This is a P1 product-trust and proof-path blocker for coilysiren/inbox#302. Resolve or classify it before selecting #183 for the recorded Ward run.

## Incident The public Eco World page was observed showing pollution numbers that had not changed for more than 12 hours on 2026-07-29. The issue is not yet classified. The stale value may come from the Eco source observation, backend fetch or cache behavior, browser page-load behavior, or incorrect timestamp presentation. Do not assume the data plane or fix before collecting evidence. ## Outcome Identify the layer that stopped advancing, restore truthful current behavior where a defect exists, and prevent the page from implying that stale pollution data is current. Freshness presentation work belongs in https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/183. This issue owns the current stale-data diagnosis and any smallest necessary repair. ## Required evidence * Record the pollution value and every available source-observation, backend-fetch, cache, and browser-load timestamp without recording private player data. * Compare the public page with the corresponding read-only API or MCP response. * Determine whether a normal page reload changes the value or timestamp. * Classify the failure as source data, backend ingestion, cache behavior, browser behavior, presentation, or an external live-system dependency. * If the defect is repository-local, prove one fix locally before pushing. * If diagnosis or verification requires a live mutation, stop at an interactive operator handoff with the exact action and expected evidence. * Do not probe CI or production through repeated speculative pushes. ## Acceptance criteria * The issue records which layer was stale and the evidence supporting that classification. * A repository defect has focused tests covering the stale path. * A live-only defect has an explicit operator action and verification result. * The page either advances on its truthful cadence or visibly reports that the pollution data is stale or unknown. * The result does not borrow the world snapshot timestamp for a different data plane. * Current validation passes, and any production verification names the deployed image and observed timestamp behavior. ## Relationship to the campaign This is a P1 product-trust and proof-path blocker for https://forgejo.coilysiren.me/coilysiren/inbox/issues/302. Resolve or classify it before selecting https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/183 for the recorded Ward run.
coilyco-ops commented 2026-08-01 20:54:04 +00:00
Member
Copy link

Ops diagnosis, 2026-08-01

Classification: source cadence plus presentation, compounded by deployment lag. Current evidence does not support a backend ingestion or cache outage.

Observed evidence

  • The public climate endpoint returned a backend fetch time of 20:19:34 UTC. A request after cache expiry returned 20:21:54 UTC. An immediate repeat reused 20:21:54 UTC. This matches the intended lazy 60-second backend cache.
  • The live cycle reported day 28. The latest TotalGroundPollution point was at 2,419,200 game seconds, exactly game-day 28. Its value was 297.55103, different from the day-27 value of 243.71794. The source series was advancing at its observed one-point-per-game-day interval.
  • The repository dataset survey describes TotalGroundPollution as a daily PPM series. The backend parser retains point times and values but discards the source Unit and Interval metadata.
  • The World page fetches climate once when it mounts and does not poll. An open page can therefore remain unchanged after the source advances. A reload reaches the same public endpoint, but the exact visual reload could not be observed because no interactive browser was connected during this run.
  • Production is healthy at one ready replica, but it is pinned to image 381023e. That image predates the issue-183 freshness UI. The deployed JavaScript bundle contains the Ground pollution surface but not the Snapshot fetched disclosure.
  • Current main contains the issue-183 disclosure, but it still hardcodes the source observation timestamp as unavailable even though the series carries the latest game-time sample.
  • Both the deployed image and current main append a percent sign to TotalGroundPollution. That source is documented as PPM. Percentage is appropriate only for the world-layer fallback.

Competing models

  • Source stopped advancing: weakened. The source has distinct day-27 and day-28 points, and the latest point matches the current game day.
  • Backend ingestion or cache stuck: weakened. The backend fetch timestamp advanced after expiry and reused only within the configured cache window.
  • Browser cache defect: not demonstrated. The confirmed browser behavior is mount-only fetching, which can leave a long-open page stale without any browser-cache failure.
  • Presentation defect: confirmed. The page does not expose the source cadence or latest game-time sample, and it renders the dataset with the wrong unit.
  • Deployment lag: confirmed. Production does not contain the already-landed issue-183 freshness disclosure.

Engineering handoff

Engineering owns the repository repair:

  • Preserve source Unit and Interval metadata alongside each climate series.
  • Expose the latest source game-time sample and a source-specific cadence.
  • Render TotalGroundPollution in PPM and reserve percentage for the world-layer fallback.
  • Classify staleness against source game time rather than backend fetch time.
  • Cover current, behind-current-game-day, unknown-cadence, cached, and fallback-unit paths with focused tests.
  • Coordinate the broader mount-only page audit through #201.

Ops follow-through

After Engineering lands and the exact image is published, Ops will roll that image, record the deployed image, reload behavior, source sample, backend fetch time, and browser load time, then verify that the page either advances on the truthful cadence or visibly reports stale or unknown.

Keep this issue open through repository repair and live verification.

## Ops diagnosis, 2026-08-01 Classification: source cadence plus presentation, compounded by deployment lag. Current evidence does not support a backend ingestion or cache outage. ## Observed evidence * The public climate endpoint returned a backend fetch time of 20:19:34 UTC. A request after cache expiry returned 20:21:54 UTC. An immediate repeat reused 20:21:54 UTC. This matches the intended lazy 60-second backend cache. * The live cycle reported day 28. The latest TotalGroundPollution point was at 2,419,200 game seconds, exactly game-day 28. Its value was 297.55103, different from the day-27 value of 243.71794. The source series was advancing at its observed one-point-per-game-day interval. * The repository dataset survey describes TotalGroundPollution as a daily PPM series. The backend parser retains point times and values but discards the source Unit and Interval metadata. * The World page fetches climate once when it mounts and does not poll. An open page can therefore remain unchanged after the source advances. A reload reaches the same public endpoint, but the exact visual reload could not be observed because no interactive browser was connected during this run. * Production is healthy at one ready replica, but it is pinned to image 381023e. That image predates the issue-183 freshness UI. The deployed JavaScript bundle contains the Ground pollution surface but not the Snapshot fetched disclosure. * Current main contains the issue-183 disclosure, but it still hardcodes the source observation timestamp as unavailable even though the series carries the latest game-time sample. * Both the deployed image and current main append a percent sign to TotalGroundPollution. That source is documented as PPM. Percentage is appropriate only for the world-layer fallback. ## Competing models * Source stopped advancing: weakened. The source has distinct day-27 and day-28 points, and the latest point matches the current game day. * Backend ingestion or cache stuck: weakened. The backend fetch timestamp advanced after expiry and reused only within the configured cache window. * Browser cache defect: not demonstrated. The confirmed browser behavior is mount-only fetching, which can leave a long-open page stale without any browser-cache failure. * Presentation defect: confirmed. The page does not expose the source cadence or latest game-time sample, and it renders the dataset with the wrong unit. * Deployment lag: confirmed. Production does not contain the already-landed issue-183 freshness disclosure. ## Engineering handoff Engineering owns the repository repair: * Preserve source Unit and Interval metadata alongside each climate series. * Expose the latest source game-time sample and a source-specific cadence. * Render TotalGroundPollution in PPM and reserve percentage for the world-layer fallback. * Classify staleness against source game time rather than backend fetch time. * Cover current, behind-current-game-day, unknown-cadence, cached, and fallback-unit paths with focused tests. * Coordinate the broader mount-only page audit through https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/201. ## Ops follow-through After Engineering lands and the exact image is published, Ops will roll that image, record the deployed image, reload behavior, source sample, backend fetch time, and browser load time, then verify that the page either advances on the truthful cadence or visibly reports stale or unknown. Keep this issue open through repository repair and live verification.
coilyco-ops commented 2026-08-01 21:26:07 +00:00
Member
Copy link

Engineering repair landed

Repository classification remains source cadence plus presentation, with deployment lag still awaiting Ops verification. The observed source and backend cache were advancing. The repository defect was that the climate parser discarded source Unit and Interval, the World page treated dataset pollution as percent, and source freshness could not be classified against game time.

Commit c72f954 landed on canonical main.

Repair

  • The dataset parser now preserves source unit and interval metadata for every selected climate series.
  • Each headline climate KPI exposes its latest source game-time sample, source cadence, current game day, cadence lag, and current, stale, or unknown state.
  • Freshness compares the source-series clock with the current game day. The backend fetch timestamp remains a separate cache timestamp.
  • TotalGroundPollution now renders as PPM in both the MCP result and World page. Percent is reserved for the world-layer fallback.
  • The World page visibly warns when pollution is behind its source cadence. Missing cadence remains explicitly unknown.
  • Focused tests cover current, behind-current-game-day, unknown-cadence, cached, and fallback-unit paths.

Local proof

  • ward exec test: 578 passed.
  • ward exec frontend-test: 158 passed.
  • ward exec lint, ward exec frontend-lint, and ward exec frontend-build: passed.
  • ward exec precommit: every managed hook passed.

Ops checkpoint

Keep this issue open. Once image forgejo.coilysiren.me/coilyco-gaming/eco-app:c72f954252d52a4ffc9218847f715d721f017ebd is published, Ops owns the rollout and live verification.

Ops should record the deployed image, reload the World page, and compare it with /preview/get_eco_climate.json. Expected evidence is the pollution value and PPM unit, source sample game time, source interval, freshness state, backend fetch time, browser load time, and normal reload behavior. A source sample on the current game day should read current. A sample at least one cadence behind should read stale. Missing cadence should read unknown.

## Engineering repair landed Repository classification remains **source cadence plus presentation**, with deployment lag still awaiting Ops verification. The observed source and backend cache were advancing. The repository defect was that the climate parser discarded source `Unit` and `Interval`, the World page treated dataset pollution as percent, and source freshness could not be classified against game time. Commit [`c72f954`](https://forgejo.coilysiren.me/coilyco-gaming/eco-app/commit/c72f954252d52a4ffc9218847f715d721f017ebd) landed on canonical `main`. ## Repair * The dataset parser now preserves source unit and interval metadata for every selected climate series. * Each headline climate KPI exposes its latest source game-time sample, source cadence, current game day, cadence lag, and `current`, `stale`, or `unknown` state. * Freshness compares the source-series clock with the current game day. The backend fetch timestamp remains a separate cache timestamp. * `TotalGroundPollution` now renders as PPM in both the MCP result and World page. Percent is reserved for the world-layer fallback. * The World page visibly warns when pollution is behind its source cadence. Missing cadence remains explicitly unknown. * Focused tests cover current, behind-current-game-day, unknown-cadence, cached, and fallback-unit paths. ## Local proof * `ward exec test`: 578 passed. * `ward exec frontend-test`: 158 passed. * `ward exec lint`, `ward exec frontend-lint`, and `ward exec frontend-build`: passed. * `ward exec precommit`: every managed hook passed. ## Ops checkpoint Keep this issue open. Once image `forgejo.coilysiren.me/coilyco-gaming/eco-app:c72f954252d52a4ffc9218847f715d721f017ebd` is published, Ops owns the rollout and live verification. Ops should record the deployed image, reload the World page, and compare it with `/preview/get_eco_climate.json`. Expected evidence is the pollution value and PPM unit, source sample game time, source interval, freshness state, backend fetch time, browser load time, and normal reload behavior. A source sample on the current game day should read current. A sample at least one cadence behind should read stale. Missing cadence should read unknown.
coilyco-ops commented 2026-08-07 20:30:19 +00:00
Member
Copy link

Read-only observation, 2026-08-07 20:29 UTC

Filed from a strats session as evidence only. This is not the Ops verification and does not close the issue. Ops still owns the browser-side checks and the disposition.

The repair is live and behaving correctly.

Deployed image, from the Kubernetes API:

forgejo.coilysiren.me/coilyco-gaming/eco-app:0239532af8eaf15059a16e3be18dfbaee0d11f66

Both coilysiren-eco-app containers, coilysiren-eco-app and discord-worker, are Running with an age of 2d21h. That places the current rollout around 2026-08-04, not today.

Observed API behavior through the public Eco MCP against http://eco.coilysiren.me:3001, fetched at 2026-08-07T20:29:12Z:

  • pollution.unit is PPM, not percent. layer_summary is "0%", so percentage is correctly reserved for the world-layer fallback.
  • Every headline series carries the observation block the repair introduced. For TotalGroundPollution: latest_game_time_seconds: 2937600, latest_game_day: 34, current_game_day: 34, interval_seconds: 86400, lag_intervals: 0, freshness_state: "current", freshness_reason: "within_source_cadence".
  • TotalCO2, SeaLevel, and AverageGlobalTemperature each carry the same structure with their own units, all reporting current at zero lag.
  • Source unit and interval metadata are preserved per series, which was the parser defect.

That is the c72f954 behavior, confirmed by output rather than by tag comparison. Image 0239532 postdates the repair and contains it.

Correction to the record

This issue's 2026-08-01 Ops comment recorded production as pinned to 381023e, an image predating the freshness UI. That was true when written and has not been true since roughly 2026-08-04.

Three pinned strategy issues were repeating it as current state as recently as an hour ago, including a claim that the public page still shows pre-repair presentation. Those are being corrected. The error was quoting this issue's own history as present state without checking the cluster, which is the same class of mistake as reading issue text for code.

What Ops still owns

The API path is verified. The browser path is not, and #184's acceptance names it explicitly:

  • Reload the World page and compare against /preview/get_eco_climate.json.
  • Record browser load time and normal reload behavior.
  • Confirm the page visibly reports stale or unknown when a source sample falls at least one cadence behind. Everything currently reads current, so the stale and unknown display paths are not exercised by today's live data.

The mount-only page-fetch behavior noted in the original diagnosis also remains an open question for eco-app#201, separate from this issue.

Consequence outside this repository

coilysiren/inbox#301 was gated on credential safety, transcript redaction, and truthful Eco delivery. The first two closed on 2026-08-04 and 2026-08-05. With this repair live, the third condition appears satisfied on the API path. That issue is now closer to unblocked than its own text suggests.

## Read-only observation, 2026-08-07 20:29 UTC Filed from a strats session as evidence only. This is not the Ops verification and does not close the issue. Ops still owns the browser-side checks and the disposition. **The repair is live and behaving correctly.** Deployed image, from the Kubernetes API: ``` forgejo.coilysiren.me/coilyco-gaming/eco-app:0239532af8eaf15059a16e3be18dfbaee0d11f66 ``` Both `coilysiren-eco-app` containers, `coilysiren-eco-app` and `discord-worker`, are Running with an age of 2d21h. That places the current rollout around 2026-08-04, not today. **Observed API behavior** through the public Eco MCP against `http://eco.coilysiren.me:3001`, fetched at `2026-08-07T20:29:12Z`: * `pollution.unit` is **`PPM`**, not percent. `layer_summary` is `"0%"`, so percentage is correctly reserved for the world-layer fallback. * Every headline series carries the observation block the repair introduced. For `TotalGroundPollution`: `latest_game_time_seconds: 2937600`, `latest_game_day: 34`, `current_game_day: 34`, `interval_seconds: 86400`, `lag_intervals: 0`, `freshness_state: "current"`, `freshness_reason: "within_source_cadence"`. * `TotalCO2`, `SeaLevel`, and `AverageGlobalTemperature` each carry the same structure with their own units, all reporting `current` at zero lag. * Source unit and interval metadata are preserved per series, which was the parser defect. That is the `c72f954` behavior, confirmed by output rather than by tag comparison. Image `0239532` postdates the repair and contains it. ## Correction to the record This issue's 2026-08-01 Ops comment recorded production as pinned to `381023e`, an image predating the freshness UI. That was true when written and has not been true since roughly 2026-08-04. Three pinned strategy issues were repeating it as current state as recently as an hour ago, including a claim that the public page still shows pre-repair presentation. Those are being corrected. The error was quoting this issue's own history as present state without checking the cluster, which is the same class of mistake as reading issue text for code. ## What Ops still owns The API path is verified. The browser path is not, and #184's acceptance names it explicitly: * Reload the World page and compare against `/preview/get_eco_climate.json`. * Record browser load time and normal reload behavior. * Confirm the page visibly reports stale or unknown when a source sample falls at least one cadence behind. Everything currently reads `current`, so the stale and unknown display paths are not exercised by today's live data. The mount-only page-fetch behavior noted in the original diagnosis also remains an open question for eco-app#201, separate from this issue. ## Consequence outside this repository https://forgejo.coilysiren.me/coilysiren/inbox/issues/301 was gated on credential safety, transcript redaction, and truthful Eco delivery. The first two closed on 2026-08-04 and 2026-08-05. With this repair live, the third condition appears satisfied on the API path. That issue is now closer to unblocked than its own text suggests.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/bk79-eco-pr-ci
aos/claude/bk79-eco-instructions
aos/claude/mg96-fm
aos/claude/mg96
feat/autogen-recipe-index
recovery/2026-07-27-local-work
ward-salvage/eco-app-84ad1890
ward-salvage/eco-app-b3384b9b
ward-salvage/eco-app-0c6e4df2
ward-salvage/eco-app-b8207ef3
ward-salvage/eco-app-a58b14ad
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
headless
Safe for autonomous warded engineer carry

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
c#
Requires C# work, flagged b/c it requires a Eco server restart

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
role/ai
requires work from the AI Engineer role

  
role/creator
requires work from Content Creator role

  
role/design
requires work from the design role

  
role/director
requires work from the director role

  
role/engineer
requires work from the engineer role

  
role/exec
requires work from the exec role

  
role/human
requires a person, and specifically not an agent seat

  
role/ops
requires work from the ops role

  
role/qa
requires work from the QA role

No labels
burndown-2026-06
headless
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
c#
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/eco-app#184
No description provided.