coilyco-gaming/eco-app
Watch
2
0
Code Issues 33 Pull requests Projects Releases Packages 1 Wiki Activity Actions

Backfill complete eco-trades history into cycle-relative price data #123

New issue
Open
opened 2026-07-20 19:21:55 +00:00 by coilysiren · 5 comments
coilysiren commented 2026-07-20 19:21:55 +00:00
Owner
Copy link

Context

Eco App already has three adjacent pieces:

  • #6 shipped the row-level current-cycle CurrencyTrade and BarterTrade ledger. Its authoritative exporter timestamp is seconds since cycle start and can be converted to in-game cycle day.
  • #156 recorded the durable goal of a normalized dataset covering every item sold across server history, including the need to retain cycle context because upgrades, logistics, and currency adoption change prices.
  • #198 consumes historical observations to produce per-item price distributions.

The missing piece is acquisition of the older history that predates durable exporter backups.

The intended source here is the DiscordLink-generated eco-trades channel containing completed-trade messages. It is not a player-authored offer channel. The previous closure decision rested on that incorrect source interpretation and is superseded.

What to build

Perform a bounded, one-time backfill of the complete accessible eco-trades channel into a normalized historical-trade dataset.

Read backward to the oldest message Discord makes available, parse every supported completed-trade message, identify the cycle in which it occurred, and express its date as both an absolute timestamp and a day relative to that cycle start.

The backfill must be restartable, idempotent, provenance-preserving, and safe to reconcile with the authoritative exporter rows already used by #6.

After this historical gap is filled, normal ongoing collection should come from the authoritative trade exporter and its periodic backups. Discord ingestion is not the long-term source of truth.

Normalized record

Preserve every field that the source can support, including:

  • cycleId - stable identifier for the Eco cycle
  • cycleDay - elapsed fractional or whole day since that cycle began
  • occurredAt - original absolute timestamp
  • item - normalized Eco item identifier plus observed display name
  • quantity
  • currency
  • totalPrice
  • unitPrice
  • buyer, seller, store, and location when present
  • sourceKind - Discord historical backfill or authoritative exporter
  • sourceMessageId for Discord records
  • sourceRowIdentity for exporter records
  • parseVersion
  • Enough source provenance to audit a value without publishing raw private history

Do not silently coerce missing or ambiguous values. Preserve a bounded parse status and reason.

Cycle-relative dating

Create or reuse a reviewable cycle-start catalog.

For each imported message:

  1. Select the cycle whose start and end bounds contain the message timestamp.
  2. Store the cycle identifier and the original timestamp.
  3. Derive cycle day from elapsed time since that cycle start.
  4. Record the timezone and precision assumptions used by the conversion.
  5. Quarantine records that cannot be assigned confidently rather than attaching them to the nearest cycle.

Cross-cycle visualization must retain cycle identity. It must not silently average unlike cycles whose progression, recipes, upgrades, logistics, or currency systems differ.

Backfill mechanics

  • Page backward from the newest relevant message to the oldest accessible message.
  • Persist a checkpoint based on stable Discord message identity.
  • Make reruns safe and free of duplicate normalized rows.
  • Record the oldest reached message, message count, accepted count, rejected count, and any known pagination or retention gap.
  • Version the parser so later corrections can reprocess the source deterministically.
  • Keep unparseable messages in a bounded quarantine report with reasons.
  • Avoid retaining unrelated chat or message content.
  • Do not ingest player-authored trade offers from other channels.
  • Do not make Discord the sole durable copy of the normalized result.

Deduplication and reconciliation

Historical Discord data may overlap with authoritative exporter history.

Implement a documented matching strategy using the strongest available combination of cycle, timestamp window, item, quantity, currency, total price, buyer, seller, and store.

The result must:

  • Never double-count a Discord message on rerun.
  • Never double-count one sale merely because both Discord and the exporter observed it.
  • Prefer the authoritative exporter record where a confident match exists.
  • Retain Discord provenance as supporting evidence when records match.
  • Keep ambiguous matches visible instead of merging them silently.
  • Produce a reconciliation summary by cycle and source.

Price-data behavior

The normalized history should support #198 without manufacturing one universal price.

For each item and currency, downstream consumers should be able to derive:

  • Observed unit prices
  • Sample count
  • Trade volume
  • First and last observation
  • Cycle-relative timing
  • Median, range, percentiles, and distribution buckets
  • Source and freshness
  • Thin, stale, multimodal, and outlier-heavy states

Currencies remain separate unless an explicit, evidence-backed conversion model exists.

Acceptance criteria

  • The importer reaches the oldest accessible eco-trades message or records the exact verified cutoff and reason it could not go farther.
  • Import progress is checkpointed and can resume without starting over.
  • Repeating the full import produces no duplicate normalized records.
  • Supported completed-trade messages produce normalized records with absolute timestamp, cycle identifier, and cycle day.
  • Cycle-start mappings and conversion assumptions are versioned and reviewable.
  • Unsupported, malformed, or ambiguous messages are counted and reported without aborting the entire backfill.
  • Discord records are reconciled against overlapping authoritative exporter rows, with exporter data preferred for confident matches.
  • A reconciliation report gives totals by cycle, source, accepted records, rejected records, duplicates, and ambiguous overlaps.
  • Price-history queries can consume the normalized records while preserving currency and cycle boundaries.
  • Tests cover pagination, restart, deduplication, cycle-boundary assignment, malformed messages, ambiguous matches, and exporter overlap.
  • Documentation distinguishes the one-time Discord backfill from ongoing authoritative exporter capture.
  • No unrelated Discord messages or raw private channel archive are published.

State and safety boundary

  • Read-only access to Discord history and existing Eco data.
  • No Discord message creation, editing, deletion, reactions, or moderation.
  • No game-server or store mutation.
  • No automated repricing.
  • No claim that historical prices are directly comparable without cycle context.
  • No indefinite Discord ingestion job after the bounded backfill succeeds.

Deliverables

  • Restartable one-time importer
  • Versioned cycle-start catalog
  • Normalized historical-trade schema
  • Deduplication and reconciliation logic
  • Import and reconciliation report
  • Tests and operator documentation
  • Durable dataset handoff to #156
  • Query support needed by #198

Related

  • Builds on #6.
  • Supplies the historical acquisition needed by #156.
  • Feeds #198.
  • Provides older-cycle context adjacent to #199.

Verified live preflight

Read-only verification on 2026-08-01 established:

  • The Eco App application deployment is available.
  • The separate Eco App Discord slash-command worker is available, but it is scoped to the dedicated eco-app channel. It is not the historical-ingestion transport.
  • The read-only Discord MCP deployment is available and its required ExternalSecrets are synced.
  • The Discord MCP can discover and read eco-trades.
  • The source messages are rich embeds authored by the Eco bot and represent completed trades.
  • A trade embed may contain either Bought or Sold data and may contain multiple item lines in one transaction.
  • The oldest directly accessible message is dated 2024-11-09.
  • The channel currently contains 137,757 messages, of which 137,745 are attributed to the Eco bot.
  • Channel-history pagination returns at most 100 messages per request. A complete pass therefore requires at least 1,378 requests before retries or reconciliation.
  • Discord supplies dynamic rate-limit evidence. The importer must obey returned retry timing rather than assume one fixed sleep interval.
  • The live Eco market surface currently sees 19,205 trade events, but 18,644 older events have collapsed into 2,663 hourly exporter rollups. Those rollups do not retain trustworthy item, party, store, or unit-price attribution. This is the specific history gap the Discord backfill fills.
  • The Eco App namespace currently has no PersistentVolumeClaim, database service, CronJob, or one-shot backfill Job.

These observations establish source availability and need. They do not mean the importer or durable storage is already deployed.

Selected end-to-end architecture

Source transport

Use the already-deployed read-only Discord MCP as the sole Discord transport.

  • Do not reuse the Eco App slash-command worker.
  • Do not inject another Discord bot token into Eco App.
  • The importer calls the internal Discord MCP service from its one-shot Job.
  • The importer paginates the bounded channel-history read with stable message checkpoints.
  • Discord API errors and rate-limit responses remain visible in the import-run report.
  • The importer accepts only the named eco-trades channel and expected Eco bot author. Any other channel or author fails closed or enters quarantine.

This keeps Discord credentials and permissions in the existing dedicated reader boundary.

Application code ownership

The Eco App repository owns:

  • The normalized historical-trade schema and migrations.
  • A non-interactive importer entrypoint in the same immutable image as Eco App.
  • The Discord embed parser.
  • Checkpoint, idempotency, parser-version, quarantine, and reconciliation behavior.
  • The read path that merges active historical observations with authoritative current exporter detail.
  • Unit, integration, and fixture tests.
  • Operator documentation describing dry-run, import, promotion, verification, and rollback.

The importer must be runnable with an explicit run identifier and support:

  • dryRun - parse and report without durable trade-observation writes.
  • beforeMessage or checkpoint resume.
  • maxMessages for bounded verification samples.
  • fullBackfill for the attended production run.
  • reconcileOnly to rerun matching without repaging Discord.
  • activateRun only after acceptance evidence passes.

Operational input ownership

The app owns the cycle-window input schema. Server-specific cycle start and end values are operational inputs owned by Eco Ops, not generic constants embedded in Eco App source.

The cycle catalog must provide:

  • Stable cycle identifier
  • Start timestamp
  • End timestamp when known
  • Display label
  • Eco version or ruleset label when known
  • Provenance for the boundary
  • Review status

The importer copies the exact catalog version or checksum into each import run. Missing or overlapping cycle windows fail closed before the full run.

Durable storage

Use a dedicated SQLite database on a new Eco App PersistentVolumeClaim as the initial storage design.

Rationale:

  • Approximately 138,000 source messages and their normalized line items are comfortably within SQLite scale.
  • Eco App is a single-replica, single-node application.
  • A new PostgreSQL service would add an unnecessary permanent operational component for a bounded backfill.
  • SQLite supports transactional imports, stable uniqueness constraints, and online backup.

Deployment requirements:

  • Add a 1 Gi expandable local persistent claim owned by the Eco App deployment surface.
  • Mount the claim at the application data directory in both the Eco App deployment and the one-shot Job.
  • Use WAL mode and explicit transactions.
  • Give every import run a run-scoped staging state.
  • Keep imported observations inactive until an operator promotes a verified run.
  • Preserve the prior active dataset during a new run.
  • Refuse a full run when available space or a verified backup target is insufficient.
  • The deployment must remain functional with an empty database and render explicit no-history states.

Minimum durable entities:

  • cycleWindow
  • importRun
  • importCheckpoint
  • sourceMessage
  • tradeTransaction
  • tradeLine
  • reconciliationMatch
  • quarantineRecord
  • activeDatasetVersion

Stable Discord message identity prevents duplicate source ingestion. A trade-line identity derived from source message plus line position prevents duplicate line items.

Deployment ownership

The deploy repository owns:

  • The Eco App PersistentVolumeClaim and application mount.
  • A tracked one-shot Kubernetes Job template using the same immutable Eco App image pin as the deployment.
  • Resource requests and limits, restart policy, failure backoff, and node placement.
  • Internal Discord MCP service discovery.
  • Rollout and read-only status evidence.

The Job is operator-triggered. It must not run from normal CD, pod startup, a recurring CronJob, or every Eco App rollout.

The Job may remain as a tracked reusable manifest, but no active backfill workload remains after successful completion.

Role handoff

Engineer

The engineer:

  • Implements the importer, schema, migrations, queries, fixtures, and tests.
  • Uses synthetic and sanitized message fixtures.
  • Proves restart, idempotency, malformed-input, multiple-line, cycle-boundary, and reconciliation behavior locally.
  • Adds the deployment inputs needed by the established Eco App deploy pattern.
  • Does not read live Discord or run the production backfill.

Ops

Ops:

  • Verifies the live Discord MCP and channel boundary.
  • Supplies and reviews the cycle catalog.
  • Selects and verifies the private backup target.
  • Observes the storage and Job rollout.
  • Runs the bounded dry run.
  • Reviews parser coverage and quarantine samples.
  • Authorizes and runs the full backfill.
  • Promotes the accepted import run.
  • Verifies application queries and player-facing evidence.
  • Records the final cutoff, totals, backup location, and rollback evidence.
  • Deletes the completed Job object while retaining the tracked template and durable data.

Advisor

Advisor returns only if live evidence creates a decision question, such as irreconcilable cross-cycle semantics, ambiguous message classes, or unacceptable parser loss.

Attended execution sequence

  1. Land application support. Publish an immutable Eco App image containing the importer, storage migration, and merged query path.
  2. Land storage support. Add the persistent claim and application mount through the established Eco App deployment path.
  3. Verify steady state. Confirm the existing app remains healthy with an empty historical database.
  4. Create cycle input. Ops supplies a reviewed, non-overlapping cycle catalog covering the historical message range.
  5. Verify backup. Ops names a private backup destination and demonstrates that a SQLite backup can be created, read, and restored. No full run proceeds without this evidence.
  6. Run a bounded dry run. Parse at least 1,000 messages spanning old and current formats with no durable observation writes.
  7. Review the dry-run report. Confirm expected author and channel, line-item math, currency parsing, cycle assignment, parser coverage, quarantine reasons, and estimated final storage.
  8. Launch the one-shot Job. Run the full paginated import with checkpointing and dynamic rate-limit handling.
  9. Resume when needed. A failure or rate limit resumes from the last committed message checkpoint without duplicating prior rows.
  10. Reconcile overlap. Compare Discord observations with detailed authoritative exporter rows and produce confident, ambiguous, and Discord-only counts.
  11. Back up the completed database. Produce and verify a versioned backup before activation.
  12. Promote the import run. Mark the accepted dataset version active transactionally.
  13. Verify the consumer. Confirm Eco App returns historical per-item and per-currency observations with cycle identity, source, sample size, and degraded states.
  14. Verify downstream presentation. Reopen or replace #198 only after the deployed dataset proves the distribution input is usable.
  15. Retire execution state. Delete the completed Kubernetes Job object. Preserve the Job template, checkpoint report, active database, and verified backup.

Dry-run and full-run evidence

Every run report must record:

  • Import run identifier
  • Application image digest
  • Parser version
  • Cycle-catalog checksum
  • Started and completed timestamps
  • First and last source-message timestamps
  • Source messages fetched
  • Expected-author messages
  • Transactions parsed
  • Trade lines parsed
  • Quarantined messages and grouped reasons
  • Duplicate source messages
  • Reconciled exporter matches
  • Ambiguous overlaps
  • Discord-only observations
  • Per-cycle counts
  • Per-currency counts
  • Database size
  • Last committed checkpoint
  • Active or inactive state
  • Backup verification result

The report contains aggregate evidence and bounded public-safe examples. It does not publish the raw Discord archive.

Failure and rollback behavior

  • A dry run cannot activate or alter historical observations.
  • The full importer commits checkpoints and normalized batches transactionally.
  • A failed batch rolls back without advancing its checkpoint.
  • A new run remains inactive until explicit promotion.
  • Promotion changes only the active dataset version.
  • Rollback selects the previous active version and restarts or refreshes the application read path.
  • Deleting a failed Job never deletes the persistent claim.
  • Schema migration failure leaves the previous readable schema and dataset untouched.
  • Unknown cycle assignment, parse loss above the accepted threshold, reconciliation anomalies, missing backup evidence, or storage exhaustion stop promotion.
  • The Discord source is never modified during recovery.

End-to-end definition of done

This issue is complete only when:

  • The deployed Discord reader can still access the verified source boundary.
  • Durable Eco App storage is deployed, mounted, and survives an application rollout.
  • The importer image and tracked one-shot Job template are landed.
  • A reviewed cycle catalog covers every promoted observation.
  • The 1,000-message dry run passes the documented review.
  • The full accessible history is imported or an exact Discord cutoff is recorded.
  • Rerunning from the initial checkpoint creates zero duplicate source messages or trade lines.
  • Every accepted trade line has source provenance, absolute time, cycle identity, and cycle day.
  • Quarantine totals and reasons are reviewed and accepted.
  • Exporter overlap is reconciled without double counting.
  • A completed database backup is verified by opening or restoring it.
  • One import run is explicitly promoted as active.
  • The deployed Eco App query path returns the promoted historical observations.
  • Existing current-cycle exporter behavior remains healthy.
  • A representative item and currency produce correct cycle-relative price evidence.
  • Empty, thin, incompatible-cycle, and unavailable-history states remain honest.
  • The final run report, cutoff, image digest, catalog checksum, and backup evidence are recorded on this issue.
  • The completed Job object is removed and no recurring Discord ingestion workload remains.

Explicit boundary with downstream issues

#123 ends when the historical dataset is durably deployed and queryable through Eco App.

#156 remains the durable dataset and future-backup product concern. #198 remains the price-distribution visualization concern. Both are currently closed and must be reopened or replaced only when #123 returns evidence proving their required input exists.

Execution issues

This issue remains the end-to-end umbrella and application work package.

  • Deployment - deploy #295 owns the Eco App PersistentVolumeClaim, application mount, and operator-triggered one-shot Job template.
  • Operations - eco-ops #70 owns the cycle catalog, backup verification, attended dry run, full import, activation, rollback evidence, and Job cleanup.

Each child issue closes only its repository or role boundary. Final deployment, import, reconciliation, and activation evidence returns here before #123 closes.

## Context Eco App already has three adjacent pieces: * #6 shipped the row-level current-cycle CurrencyTrade and BarterTrade ledger. Its authoritative exporter timestamp is seconds since cycle start and can be converted to in-game cycle day. * #156 recorded the durable goal of a normalized dataset covering every item sold across server history, including the need to retain cycle context because upgrades, logistics, and currency adoption change prices. * #198 consumes historical observations to produce per-item price distributions. The missing piece is acquisition of the older history that predates durable exporter backups. The intended source here is the DiscordLink-generated **eco-trades** channel containing completed-trade messages. It is not a player-authored offer channel. The previous closure decision rested on that incorrect source interpretation and is superseded. ## What to build Perform a bounded, one-time backfill of the complete accessible **eco-trades** channel into a normalized historical-trade dataset. Read backward to the oldest message Discord makes available, parse every supported completed-trade message, identify the cycle in which it occurred, and express its date as both an absolute timestamp and a day relative to that cycle start. The backfill must be restartable, idempotent, provenance-preserving, and safe to reconcile with the authoritative exporter rows already used by #6. After this historical gap is filled, normal ongoing collection should come from the authoritative trade exporter and its periodic backups. Discord ingestion is not the long-term source of truth. ## Normalized record Preserve every field that the source can support, including: * **cycleId** - stable identifier for the Eco cycle * **cycleDay** - elapsed fractional or whole day since that cycle began * **occurredAt** - original absolute timestamp * **item** - normalized Eco item identifier plus observed display name * **quantity** * **currency** * **totalPrice** * **unitPrice** * **buyer**, **seller**, **store**, and **location** when present * **sourceKind** - Discord historical backfill or authoritative exporter * **sourceMessageId** for Discord records * **sourceRowIdentity** for exporter records * **parseVersion** * Enough source provenance to audit a value without publishing raw private history Do not silently coerce missing or ambiguous values. Preserve a bounded parse status and reason. ## Cycle-relative dating Create or reuse a reviewable cycle-start catalog. For each imported message: 1. Select the cycle whose start and end bounds contain the message timestamp. 2. Store the cycle identifier and the original timestamp. 3. Derive cycle day from elapsed time since that cycle start. 4. Record the timezone and precision assumptions used by the conversion. 5. Quarantine records that cannot be assigned confidently rather than attaching them to the nearest cycle. Cross-cycle visualization must retain cycle identity. It must not silently average unlike cycles whose progression, recipes, upgrades, logistics, or currency systems differ. ## Backfill mechanics * Page backward from the newest relevant message to the oldest accessible message. * Persist a checkpoint based on stable Discord message identity. * Make reruns safe and free of duplicate normalized rows. * Record the oldest reached message, message count, accepted count, rejected count, and any known pagination or retention gap. * Version the parser so later corrections can reprocess the source deterministically. * Keep unparseable messages in a bounded quarantine report with reasons. * Avoid retaining unrelated chat or message content. * Do not ingest player-authored trade offers from other channels. * Do not make Discord the sole durable copy of the normalized result. ## Deduplication and reconciliation Historical Discord data may overlap with authoritative exporter history. Implement a documented matching strategy using the strongest available combination of cycle, timestamp window, item, quantity, currency, total price, buyer, seller, and store. The result must: * Never double-count a Discord message on rerun. * Never double-count one sale merely because both Discord and the exporter observed it. * Prefer the authoritative exporter record where a confident match exists. * Retain Discord provenance as supporting evidence when records match. * Keep ambiguous matches visible instead of merging them silently. * Produce a reconciliation summary by cycle and source. ## Price-data behavior The normalized history should support #198 without manufacturing one universal price. For each item and currency, downstream consumers should be able to derive: * Observed unit prices * Sample count * Trade volume * First and last observation * Cycle-relative timing * Median, range, percentiles, and distribution buckets * Source and freshness * Thin, stale, multimodal, and outlier-heavy states Currencies remain separate unless an explicit, evidence-backed conversion model exists. ## Acceptance criteria * [ ] The importer reaches the oldest accessible **eco-trades** message or records the exact verified cutoff and reason it could not go farther. * [ ] Import progress is checkpointed and can resume without starting over. * [ ] Repeating the full import produces no duplicate normalized records. * [ ] Supported completed-trade messages produce normalized records with absolute timestamp, cycle identifier, and cycle day. * [ ] Cycle-start mappings and conversion assumptions are versioned and reviewable. * [ ] Unsupported, malformed, or ambiguous messages are counted and reported without aborting the entire backfill. * [ ] Discord records are reconciled against overlapping authoritative exporter rows, with exporter data preferred for confident matches. * [ ] A reconciliation report gives totals by cycle, source, accepted records, rejected records, duplicates, and ambiguous overlaps. * [ ] Price-history queries can consume the normalized records while preserving currency and cycle boundaries. * [ ] Tests cover pagination, restart, deduplication, cycle-boundary assignment, malformed messages, ambiguous matches, and exporter overlap. * [ ] Documentation distinguishes the one-time Discord backfill from ongoing authoritative exporter capture. * [ ] No unrelated Discord messages or raw private channel archive are published. ## State and safety boundary * Read-only access to Discord history and existing Eco data. * No Discord message creation, editing, deletion, reactions, or moderation. * No game-server or store mutation. * No automated repricing. * No claim that historical prices are directly comparable without cycle context. * No indefinite Discord ingestion job after the bounded backfill succeeds. ## Deliverables * Restartable one-time importer * Versioned cycle-start catalog * Normalized historical-trade schema * Deduplication and reconciliation logic * Import and reconciliation report * Tests and operator documentation * Durable dataset handoff to #156 * Query support needed by #198 ## Related * Builds on #6. * Supplies the historical acquisition needed by #156. * Feeds #198. * Provides older-cycle context adjacent to #199. ## Verified live preflight Read-only verification on 2026-08-01 established: * The Eco App application deployment is available. * The separate Eco App Discord slash-command worker is available, but it is scoped to the dedicated **eco-app** channel. It is not the historical-ingestion transport. * The read-only Discord MCP deployment is available and its required ExternalSecrets are synced. * The Discord MCP can discover and read **eco-trades**. * The source messages are rich embeds authored by the Eco bot and represent completed trades. * A trade embed may contain either Bought or Sold data and may contain multiple item lines in one transaction. * The oldest directly accessible message is dated 2024-11-09. * The channel currently contains 137,757 messages, of which 137,745 are attributed to the Eco bot. * Channel-history pagination returns at most 100 messages per request. A complete pass therefore requires at least 1,378 requests before retries or reconciliation. * Discord supplies dynamic rate-limit evidence. The importer must obey returned retry timing rather than assume one fixed sleep interval. * The live Eco market surface currently sees 19,205 trade events, but 18,644 older events have collapsed into 2,663 hourly exporter rollups. Those rollups do not retain trustworthy item, party, store, or unit-price attribution. This is the specific history gap the Discord backfill fills. * The Eco App namespace currently has no PersistentVolumeClaim, database service, CronJob, or one-shot backfill Job. These observations establish source availability and need. They do not mean the importer or durable storage is already deployed. ## Selected end-to-end architecture ### Source transport Use the already-deployed read-only Discord MCP as the sole Discord transport. * Do not reuse the Eco App slash-command worker. * Do not inject another Discord bot token into Eco App. * The importer calls the internal Discord MCP service from its one-shot Job. * The importer paginates the bounded channel-history read with stable message checkpoints. * Discord API errors and rate-limit responses remain visible in the import-run report. * The importer accepts only the named **eco-trades** channel and expected Eco bot author. Any other channel or author fails closed or enters quarantine. This keeps Discord credentials and permissions in the existing dedicated reader boundary. ### Application code ownership The Eco App repository owns: * The normalized historical-trade schema and migrations. * A non-interactive importer entrypoint in the same immutable image as Eco App. * The Discord embed parser. * Checkpoint, idempotency, parser-version, quarantine, and reconciliation behavior. * The read path that merges active historical observations with authoritative current exporter detail. * Unit, integration, and fixture tests. * Operator documentation describing dry-run, import, promotion, verification, and rollback. The importer must be runnable with an explicit run identifier and support: * **dryRun** - parse and report without durable trade-observation writes. * **beforeMessage** or checkpoint resume. * **maxMessages** for bounded verification samples. * **fullBackfill** for the attended production run. * **reconcileOnly** to rerun matching without repaging Discord. * **activateRun** only after acceptance evidence passes. ### Operational input ownership The app owns the cycle-window input schema. Server-specific cycle start and end values are operational inputs owned by Eco Ops, not generic constants embedded in Eco App source. The cycle catalog must provide: * Stable cycle identifier * Start timestamp * End timestamp when known * Display label * Eco version or ruleset label when known * Provenance for the boundary * Review status The importer copies the exact catalog version or checksum into each import run. Missing or overlapping cycle windows fail closed before the full run. ### Durable storage Use a dedicated SQLite database on a new Eco App PersistentVolumeClaim as the initial storage design. Rationale: * Approximately 138,000 source messages and their normalized line items are comfortably within SQLite scale. * Eco App is a single-replica, single-node application. * A new PostgreSQL service would add an unnecessary permanent operational component for a bounded backfill. * SQLite supports transactional imports, stable uniqueness constraints, and online backup. Deployment requirements: * Add a 1 Gi expandable local persistent claim owned by the Eco App deployment surface. * Mount the claim at the application data directory in both the Eco App deployment and the one-shot Job. * Use WAL mode and explicit transactions. * Give every import run a run-scoped staging state. * Keep imported observations inactive until an operator promotes a verified run. * Preserve the prior active dataset during a new run. * Refuse a full run when available space or a verified backup target is insufficient. * The deployment must remain functional with an empty database and render explicit no-history states. Minimum durable entities: * **cycleWindow** * **importRun** * **importCheckpoint** * **sourceMessage** * **tradeTransaction** * **tradeLine** * **reconciliationMatch** * **quarantineRecord** * **activeDatasetVersion** Stable Discord message identity prevents duplicate source ingestion. A trade-line identity derived from source message plus line position prevents duplicate line items. ### Deployment ownership The deploy repository owns: * The Eco App PersistentVolumeClaim and application mount. * A tracked one-shot Kubernetes Job template using the same immutable Eco App image pin as the deployment. * Resource requests and limits, restart policy, failure backoff, and node placement. * Internal Discord MCP service discovery. * Rollout and read-only status evidence. The Job is operator-triggered. It must not run from normal CD, pod startup, a recurring CronJob, or every Eco App rollout. The Job may remain as a tracked reusable manifest, but no active backfill workload remains after successful completion. ## Role handoff ### Engineer The engineer: * Implements the importer, schema, migrations, queries, fixtures, and tests. * Uses synthetic and sanitized message fixtures. * Proves restart, idempotency, malformed-input, multiple-line, cycle-boundary, and reconciliation behavior locally. * Adds the deployment inputs needed by the established Eco App deploy pattern. * Does not read live Discord or run the production backfill. ### Ops Ops: * Verifies the live Discord MCP and channel boundary. * Supplies and reviews the cycle catalog. * Selects and verifies the private backup target. * Observes the storage and Job rollout. * Runs the bounded dry run. * Reviews parser coverage and quarantine samples. * Authorizes and runs the full backfill. * Promotes the accepted import run. * Verifies application queries and player-facing evidence. * Records the final cutoff, totals, backup location, and rollback evidence. * Deletes the completed Job object while retaining the tracked template and durable data. ### Advisor Advisor returns only if live evidence creates a decision question, such as irreconcilable cross-cycle semantics, ambiguous message classes, or unacceptable parser loss. ## Attended execution sequence 1. **Land application support.** Publish an immutable Eco App image containing the importer, storage migration, and merged query path. 2. **Land storage support.** Add the persistent claim and application mount through the established Eco App deployment path. 3. **Verify steady state.** Confirm the existing app remains healthy with an empty historical database. 4. **Create cycle input.** Ops supplies a reviewed, non-overlapping cycle catalog covering the historical message range. 5. **Verify backup.** Ops names a private backup destination and demonstrates that a SQLite backup can be created, read, and restored. No full run proceeds without this evidence. 6. **Run a bounded dry run.** Parse at least 1,000 messages spanning old and current formats with no durable observation writes. 7. **Review the dry-run report.** Confirm expected author and channel, line-item math, currency parsing, cycle assignment, parser coverage, quarantine reasons, and estimated final storage. 8. **Launch the one-shot Job.** Run the full paginated import with checkpointing and dynamic rate-limit handling. 9. **Resume when needed.** A failure or rate limit resumes from the last committed message checkpoint without duplicating prior rows. 10. **Reconcile overlap.** Compare Discord observations with detailed authoritative exporter rows and produce confident, ambiguous, and Discord-only counts. 11. **Back up the completed database.** Produce and verify a versioned backup before activation. 12. **Promote the import run.** Mark the accepted dataset version active transactionally. 13. **Verify the consumer.** Confirm Eco App returns historical per-item and per-currency observations with cycle identity, source, sample size, and degraded states. 14. **Verify downstream presentation.** Reopen or replace #198 only after the deployed dataset proves the distribution input is usable. 15. **Retire execution state.** Delete the completed Kubernetes Job object. Preserve the Job template, checkpoint report, active database, and verified backup. ## Dry-run and full-run evidence Every run report must record: * Import run identifier * Application image digest * Parser version * Cycle-catalog checksum * Started and completed timestamps * First and last source-message timestamps * Source messages fetched * Expected-author messages * Transactions parsed * Trade lines parsed * Quarantined messages and grouped reasons * Duplicate source messages * Reconciled exporter matches * Ambiguous overlaps * Discord-only observations * Per-cycle counts * Per-currency counts * Database size * Last committed checkpoint * Active or inactive state * Backup verification result The report contains aggregate evidence and bounded public-safe examples. It does not publish the raw Discord archive. ## Failure and rollback behavior * A dry run cannot activate or alter historical observations. * The full importer commits checkpoints and normalized batches transactionally. * A failed batch rolls back without advancing its checkpoint. * A new run remains inactive until explicit promotion. * Promotion changes only the active dataset version. * Rollback selects the previous active version and restarts or refreshes the application read path. * Deleting a failed Job never deletes the persistent claim. * Schema migration failure leaves the previous readable schema and dataset untouched. * Unknown cycle assignment, parse loss above the accepted threshold, reconciliation anomalies, missing backup evidence, or storage exhaustion stop promotion. * The Discord source is never modified during recovery. ## End-to-end definition of done This issue is complete only when: * [ ] The deployed Discord reader can still access the verified source boundary. * [ ] Durable Eco App storage is deployed, mounted, and survives an application rollout. * [ ] The importer image and tracked one-shot Job template are landed. * [ ] A reviewed cycle catalog covers every promoted observation. * [ ] The 1,000-message dry run passes the documented review. * [ ] The full accessible history is imported or an exact Discord cutoff is recorded. * [ ] Rerunning from the initial checkpoint creates zero duplicate source messages or trade lines. * [ ] Every accepted trade line has source provenance, absolute time, cycle identity, and cycle day. * [ ] Quarantine totals and reasons are reviewed and accepted. * [ ] Exporter overlap is reconciled without double counting. * [ ] A completed database backup is verified by opening or restoring it. * [ ] One import run is explicitly promoted as active. * [ ] The deployed Eco App query path returns the promoted historical observations. * [ ] Existing current-cycle exporter behavior remains healthy. * [ ] A representative item and currency produce correct cycle-relative price evidence. * [ ] Empty, thin, incompatible-cycle, and unavailable-history states remain honest. * [ ] The final run report, cutoff, image digest, catalog checksum, and backup evidence are recorded on this issue. * [ ] The completed Job object is removed and no recurring Discord ingestion workload remains. ## Explicit boundary with downstream issues #123 ends when the historical dataset is durably deployed and queryable through Eco App. #156 remains the durable dataset and future-backup product concern. #198 remains the price-distribution visualization concern. Both are currently closed and must be reopened or replaced only when #123 returns evidence proving their required input exists. ## Execution issues This issue remains the end-to-end umbrella and application work package. * **Deployment** - [deploy #295](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/295) owns the Eco App PersistentVolumeClaim, application mount, and operator-triggered one-shot Job template. * **Operations** - [eco-ops #70](https://forgejo.coilysiren.me/coilyco-gaming/eco-ops/issues/70) owns the cycle catalog, backup verification, attended dry run, full import, activation, rollback evidence, and Job cleanup. Each child issue closes only its repository or role boundary. Final deployment, import, reconciliation, and activation evidence returns here before #123 closes.
coilyco-ops commented 2026-07-26 00:36:43 +00:00
Member
Copy link

Decision: do not import the Discord trade-channel backlog.

Discord posts are player-authored messages, not an authoritative ledger of completed sales. Offers can be edited, deleted, reposted, or fulfilled at a different price, and a historical export would introduce a privacy and retention contract for weak pricing evidence.

#156 now owns the durable pricing dataset from actual in-game sale records, including periodic backups and per-item distributions. That is the correct source for guidance going forward. Prior-cycle Discord history remains out of scope unless Kai later requests a separately consented archival project.

Closing this proposed Discord backfill without exporting or ingesting player messages.

Decision: do not import the Discord trade-channel backlog. Discord posts are player-authored messages, not an authoritative ledger of completed sales. Offers can be edited, deleted, reposted, or fulfilled at a different price, and a historical export would introduce a privacy and retention contract for weak pricing evidence. #156 now owns the durable pricing dataset from actual in-game sale records, including periodic backups and per-item distributions. That is the correct source for guidance going forward. Prior-cycle Discord history remains out of scope unless Kai later requests a separately consented archival project. Closing this proposed Discord backfill without exporting or ingesting player messages.
coilyco-ops changed title from Pull historical data to establish pricing guidelines to Backfill complete eco-trades history into cycle-relative price data 2026-08-01 20:23:10 +00:00
coilyco-ops commented 2026-08-01 20:23:14 +00:00
Member
Copy link

Correction to the previous closure: the intended eco-trades source is the DiscordLink-generated completed-trade feed, not a player-authored offer channel. The privacy and weak-evidence rationale therefore addressed the wrong data source.

This issue is reopened as the bounded historical acquisition job. #156 remains the durable normalized-dataset concern, and #198 remains the downstream per-item distribution and presentation concern.

Correction to the previous closure: the intended **eco-trades** source is the DiscordLink-generated completed-trade feed, not a player-authored offer channel. The privacy and weak-evidence rationale therefore addressed the wrong data source. This issue is reopened as the bounded historical acquisition job. #156 remains the durable normalized-dataset concern, and #198 remains the downstream per-item distribution and presentation concern.
coilyco-ops commented 2026-08-01 20:40:44 +00:00
Member
Copy link

End-to-end spec update: live preflight confirmed that the completed-trade feed is accessible through the deployed read-only Discord MCP, while Eco App has no durable historical store or backfill workload yet. The issue now selects that existing reader boundary, a dedicated Eco App SQLite PVC, and an operator-triggered one-shot Job. It defines application, deploy, and Eco Ops ownership, cycle-catalog inputs, dry-run and full-run evidence, reconciliation, promotion, rollback, and deployed definition-of-done criteria. The label is now interactive because the production import, backup verification, and activation require an attended Ops checkpoint.

End-to-end spec update: live preflight confirmed that the completed-trade feed is accessible through the deployed read-only Discord MCP, while Eco App has no durable historical store or backfill workload yet. The issue now selects that existing reader boundary, a dedicated Eco App SQLite PVC, and an operator-triggered one-shot Job. It defines application, deploy, and Eco Ops ownership, cycle-catalog inputs, dry-run and full-run evidence, reconciliation, promotion, rollback, and deployed definition-of-done criteria. The label is now interactive because the production import, backup verification, and activation require an attended Ops checkpoint.
coilyco-ops commented 2026-08-01 21:02:27 +00:00
Member
Copy link

Execution split created. deploy #295 owns persistent storage and the one-shot Job. eco-ops #70 owns the cycle catalog and attended import. This issue remains the application work package and end-to-end evidence umbrella.

Execution split created. [deploy #295](https://forgejo.coilysiren.me/coilyco-bridge/deploy/issues/295) owns persistent storage and the one-shot Job. [eco-ops #70](https://forgejo.coilysiren.me/coilyco-gaming/eco-ops/issues/70) owns the cycle catalog and attended import. This issue remains the application work package and end-to-end evidence umbrella.
coilyco-ops commented 2026-08-20 05:05:58 +00:00
Member
Copy link

Discord source for this work, found while sweeping #suggestions-feedback-bugs into Forgejo. Not filed separately because it is the same deliverable as this issue.

Thread // https://discord.com/channels/1300204416229441587/1530725974105395331
Opened by // Kai, 2026-07-25
Title // "Use discord link history to create pricing bell curves for each item"
Signal // no replies

The thread body is a pointer: "See primarily [the pricing thread]". So the framing lives in the parent discussion at https://discord.com/channels/1300204416229441587/1530651520150802472, which ran to 30 replies and is the busiest thread in the forum.

What the title adds to this issue

A specific output shape. Not just backfilled price history, but a distribution per item, so a seller can see where their price sits against what the item has actually traded at rather than against one calculated number.

Why the community needs it, which is the part worth carrying over

The existing tools are failing, and that is what pushed Kai toward building this:

  • eco-calc has not been updated for v13. It assumes a 40% raw input reduction where an AU3 actually gives 15%. Elizabeth: "if you're using it and accounting for upgrade related discounts with it too, you're very likely selling at a lower profit or even a loss." thejaxx, who had been following it: "So I guess I was selling at a loss." Upstream: https://github.com/aritchie05/EcoCraftingTool/issues/188
  • Eco Gnome is current on upgrades but opaque. Theo reports a 50% margin setting yielding roughly 75% margin, worsening as it goes higher. Its one clear advantage: it shows prices even when a shop is sold out.

Kai's conclusion in that thread:

This seems like a good case for... a new website to appear! One that knows our community specifically! Luckily I have just the thing

And the gap she named: "I need to populate price info tho. I can do that ... maybe later today?"

Design input from the same discussion

A bell curve is descriptive, and the community explicitly rejected prescriptive pricing. Both Kai and Octavian came out against forcing prices. That is an argument for this feature rather than against it: a distribution informs without mandating, which is the only shape that fits the stated policy.

Two open questions that affect what the curve should be computed over:

  1. Should transport and sourcing effort be visible in the data? StiFFFy pays double for copper because it is across the globe, and gets accused of distorting global prices for it. Octavian and Theo argue travel is part of the skill and should not reach the price. A raw distribution will show StiFFFy's outlier without explaining it. Consider whether biome or distance is a dimension worth carrying.
  2. Per-cycle or cross-cycle? This issue already frames the backfill as cycle-relative. Confirm the curve respects that, since mod sets and upgrade percentages differ between cycles and pooling them would produce a meaningless distribution.

Related issues filed from the same sweep

  • #313 , eco-app feature inventory against eco-calc, Eco Gnome, and uCat's calculator
  • coilyco-gaming/eco-ops#80 , the undercutting rule and pricing policy, which is the consumer of this data
  • coilyco-gaming/eco-mods#25 , boat shops not syncing to Eco Gnome, a gap in the same price-visibility surface
**Discord source for this work**, found while sweeping `#suggestions-feedback-bugs` into Forgejo. Not filed separately because it is the same deliverable as this issue. **Thread** // https://discord.com/channels/1300204416229441587/1530725974105395331 **Opened by** // Kai, 2026-07-25 **Title** // "Use discord link history to create pricing bell curves for each item" **Signal** // no replies The thread body is a pointer: "See primarily [the pricing thread]". So the framing lives in the parent discussion at https://discord.com/channels/1300204416229441587/1530651520150802472, which ran to 30 replies and is the busiest thread in the forum. ## What the title adds to this issue A specific output shape. Not just backfilled price history, but **a distribution per item**, so a seller can see where their price sits against what the item has actually traded at rather than against one calculated number. ## Why the community needs it, which is the part worth carrying over The existing tools are failing, and that is what pushed Kai toward building this: * **eco-calc has not been updated for v13.** It assumes a 40% raw input reduction where an AU3 actually gives 15%. Elizabeth: "if you're using it and accounting for upgrade related discounts with it too, you're very likely selling at a lower profit or even a loss." thejaxx, who had been following it: "So I guess I was selling at a loss." Upstream: https://github.com/aritchie05/EcoCraftingTool/issues/188 * **Eco Gnome is current on upgrades but opaque.** Theo reports a 50% margin setting yielding roughly 75% margin, worsening as it goes higher. Its one clear advantage: it shows prices even when a shop is sold out. Kai's conclusion in that thread: > This seems like a good case for... a new website to appear! One that knows our community specifically! Luckily I have just the thing And the gap she named: "I need to populate price info tho. I can do that ... maybe later today?" ## Design input from the same discussion A bell curve is descriptive, and the community explicitly rejected prescriptive pricing. Both Kai and Octavian came out against forcing prices. That is an argument for this feature rather than against it: a distribution informs without mandating, which is the only shape that fits the stated policy. Two open questions that affect what the curve should be computed over: 1. **Should transport and sourcing effort be visible in the data?** StiFFFy pays double for copper because it is across the globe, and gets accused of distorting global prices for it. Octavian and Theo argue travel is part of the skill and should not reach the price. A raw distribution will show StiFFFy's outlier without explaining it. Consider whether biome or distance is a dimension worth carrying. 2. **Per-cycle or cross-cycle?** This issue already frames the backfill as cycle-relative. Confirm the curve respects that, since mod sets and upgrade percentages differ between cycles and pooling them would produce a meaningless distribution. ## Related issues filed from the same sweep * https://forgejo.coilysiren.me/coilyco-gaming/eco-app/issues/313 , eco-app feature inventory against eco-calc, Eco Gnome, and uCat's calculator * https://forgejo.coilysiren.me/coilyco-gaming/eco-ops/issues/80 , the undercutting rule and pricing policy, which is the consumer of this data * https://forgejo.coilysiren.me/coilyco-gaming/eco-mods/issues/25 , boat shops not syncing to Eco Gnome, a gap in the same price-visibility surface
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/bk79-eco-pr-ci
aos/claude/bk79-eco-instructions
aos/claude/mg96-fm
aos/claude/mg96
feat/autogen-recipe-index
recovery/2026-07-27-local-work
ward-salvage/eco-app-84ad1890
ward-salvage/eco-app-b3384b9b
ward-salvage/eco-app-0c6e4df2
ward-salvage/eco-app-b8207ef3
ward-salvage/eco-app-a58b14ad
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
headless
Safe for autonomous warded engineer carry

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
c#
Requires C# work, flagged b/c it requires a Eco server restart

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
role/ai
requires work from the AI Engineer role

  
role/creator
requires work from Content Creator role

  
role/design
requires work from the design role

  
role/director
requires work from the director role

  
role/engineer
requires work from the engineer role

  
role/exec
requires work from the exec role

  
role/human
requires a person, and specifically not an agent seat

  
role/ops
requires work from the ops role

  
role/qa
requires work from the QA role

No labels
burndown-2026-06
headless
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
c#
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
role/ai
role/creator
role/design
role/director
role/engineer
role/exec
role/human
role/ops
role/qa
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-gaming/eco-app#123
No description provided.