coilyco-gaming/eco-app

Fork 0

ActionMapper: bound serialization breadth, move SQLite write off the action thread #28

New issue

Closed

opened 2026-06-17 07:29:43 +00:00 by coilysiren · 2 comments

coilysiren commented

2026-06-17 07:29:43 +00:00

Owner

Originally filed by @coilysiren on 2026-05-19T02:33:38Z

Problem - With eco-replay installed, a single player click on the live server (coilysiren/infrastructure#183) allocated enough memory to freeze kai-server. Idle play was fine. Two structural bugs in the recorder make this possible.

Bug 1: ActionMapper.ToRow serialization is bounded in depth but not breadth.

BodySettings sets MaxDepth = 2, which only constrains nesting. ShallowResolver short-circuits properties whose declared PropertyType.FullName is exactly one of User / ItemStack / WorldObject / Deed. That misses:

Properties typed as object, IAlias, IOwned, or any interface / base class. Runtime type doesn't match the FullName exact-match, so Newtonsoft enumerates.
Properties typed as IEnumerable<T>, IList<T>, IDictionary<,>, HashSet<T> of complex Eco entities (inventories, chunks, world-object lists, property graphs). MaxDepth=2 lets each element be fully expanded one layer.
Newtonsoft's Error handler swallows exceptions but doesn't stop enumeration. A 100k-element collection of complex objects allocates GB of intermediate strings before anything errors.

Click actions (interact / place / trade / claim) carry exactly these wide references. Idle actions don't.

Fix sketch:

In ShallowResolver, walk prop.PropertyType and its base types + implemented interfaces against SkipTypes, not just exact FullName match. Better: maintain an allow-list of safe primitive-ish property types and skip everything else.
Treat any IEnumerable (other than string and primitive collections) at any depth as a count summary: \"<n items>\", not the full enumeration. A custom JsonConverter on the resolver is cleaner than fighting DefaultContractResolver.
Hard cap body_json size after the fact: if > 16 KB after serialization, replace with {\"truncated\": true, \"action_type\": ...}.

Bug 2: SQLite insert runs synchronously on Eco's action thread.

EventStore.Insert takes writeLock and does ExecuteNonQuery from inside ActionPerformed. That's on whatever thread Eco fires ActionUtil. Under load this:

Couples the game tick to disk latency.
Serializes every action through a single mutex.
Amplifies any pathology in Bug 1 (the runaway serialization runs on the game thread too).

Fix sketch:

Channel<EventRow>(bounded, DropOldest) between ActionPerformed and a single background Task that drains the channel and batches inserts.
ActionPerformed becomes: build row, TryWrite, return. Never blocks the game thread.
The background writer can batch (transaction over 100 rows or 250 ms, whichever comes first) for better throughput.

Out of scope - the host-level memory cage already landed in coilysiren/infrastructure#183. That stops the host from freezing again, but the mod is still uninstalled until this issue closes.

Verify -

Unit tests covering a synthetic GameAction with a 10k-item IEnumerable property: ToRow returns in bounded time, body_json size is bounded.
Reinstall on kai-server, confirm clicks no longer balloon RSS, Storage/EcoReplay.db grows normally.
MemoryHigh=10G cgroup soft-cap does not get touched during normal play.

Migrated from coilyco-flight-deck/eco-replay#1 during the eco repo consolidation (coilysiren/inbox#100).

_Originally filed by @coilysiren on 2026-05-19T02:33:38Z_ **Problem** - With eco-replay installed, a single player click on the live server (coilysiren/infrastructure#183) allocated enough memory to freeze kai-server. Idle play was fine. Two structural bugs in the recorder make this possible. **Bug 1: ``ActionMapper.ToRow`` serialization is bounded in depth but not breadth.** ``BodySettings`` sets ``MaxDepth = 2``, which only constrains nesting. ``ShallowResolver`` short-circuits properties whose declared ``PropertyType.FullName`` is exactly one of ``User`` / ``ItemStack`` / ``WorldObject`` / ``Deed``. That misses: - Properties typed as ``object``, ``IAlias``, ``IOwned``, or any interface / base class. Runtime type doesn't match the ``FullName`` exact-match, so Newtonsoft enumerates. - Properties typed as ``IEnumerable<T>``, ``IList<T>``, ``IDictionary<,>``, ``HashSet<T>`` of complex Eco entities (inventories, chunks, world-object lists, property graphs). MaxDepth=2 lets each element be fully expanded one layer. - Newtonsoft's ``Error`` handler swallows exceptions but doesn't stop enumeration. A 100k-element collection of complex objects allocates GB of intermediate strings before anything errors. Click actions (interact / place / trade / claim) carry exactly these wide references. Idle actions don't. **Fix sketch:** - In ``ShallowResolver``, walk ``prop.PropertyType`` and its base types + implemented interfaces against ``SkipTypes``, not just exact ``FullName`` match. Better: maintain an allow-list of safe primitive-ish property types and skip everything else. - Treat any ``IEnumerable`` (other than ``string`` and primitive collections) at any depth as a count summary: ``\"<n items>\"``, not the full enumeration. A custom ``JsonConverter`` on the resolver is cleaner than fighting ``DefaultContractResolver``. - Hard cap ``body_json`` size after the fact: if ``> 16 KB`` after serialization, replace with ``{\"truncated\": true, \"action_type\": ...}``. **Bug 2: SQLite insert runs synchronously on Eco's action thread.** ``EventStore.Insert`` takes ``writeLock`` and does ``ExecuteNonQuery`` from inside ``ActionPerformed``. That's on whatever thread Eco fires ``ActionUtil``. Under load this: - Couples the game tick to disk latency. - Serializes every action through a single mutex. - Amplifies any pathology in Bug 1 (the runaway serialization runs on the game thread too). **Fix sketch:** - ``Channel<EventRow>(bounded, DropOldest)`` between ``ActionPerformed`` and a single background ``Task`` that drains the channel and batches inserts. - ``ActionPerformed`` becomes: build row, ``TryWrite``, return. Never blocks the game thread. - The background writer can batch (transaction over 100 rows or 250 ms, whichever comes first) for better throughput. **Out of scope** - the host-level memory cage already landed in coilysiren/infrastructure#183. That stops the host from freezing again, but the mod is still uninstalled until this issue closes. **Verify** - 1. Unit tests covering a synthetic ``GameAction`` with a 10k-item ``IEnumerable`` property: ``ToRow`` returns in bounded time, ``body_json`` size is bounded. 2. Reinstall on kai-server, confirm clicks no longer balloon RSS, ``Storage/EcoReplay.db`` grows normally. 3. ``MemoryHigh=10G`` cgroup soft-cap does not get touched during normal play. --- _Migrated from coilyco-flight-deck/eco-replay#1 during the eco repo consolidation (coilysiren/inbox#100)._

coilysiren added the

label

2026-06-17 07:29:44 +00:00

coilyco-ops commented

2026-07-05 03:20:10 +00:00

Member

🔒 Reserved by ward agent --driver claude — container engineer-claude-eco-app-28 on host KAI-DESKTOP-TOWER is carrying this issue (reserved 2026-07-05T03:20:00Z). Concurrent ward agent runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); --force overrides.

Do not comment on or edit this issue to steer the run while it is reserved. The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a new issue, dispatched fresh — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).

— Claude (she/her), via ward agent

🔒 Reserved by `ward agent --driver claude` — container `engineer-claude-eco-app-28` on host `KAI-DESKTOP-TOWER` is carrying this issue (reserved 2026-07-05T03:20:00Z). Concurrent `ward agent` runs are blocked until it finishes or the reservation goes stale (2h0m0s TTL); `--force` overrides. **Do not comment on or edit this issue to steer the run while it is reserved.** The engineer seeded the body once at launch and never re-reads it, so a comment or edit reaches only human readers, never the running engineer. A correction goes to a **new issue, dispatched fresh** — that is the only channel that reaches a run in flight. Where the forge supports it, ward locks this conversation to make that a road-block rather than a convention (ward#494).  — Claude (she/her), via `ward agent`

coilysiren referenced this issue from a commit

2026-07-05 03:32:37 +00:00

fix(replay): bound action serialization breadth, move SQLite write off the game thread

coilysiren closed this issue

2026-07-05 03:32:37 +00:00

coilysiren referenced this issue from a commit

2026-07-05 03:32:37 +00:00

Merge branch 'issue-28': bound replay serialization, async SQLite writer (#28)

coilysiren commented

2026-07-05 03:33:34 +00:00

Author

Owner

WARD-OUTCOME: done - bounded action-body serialization (allow-list resolver + 16 KB cap) and moved SQLite writes to a background bounded-channel batched writer; landed on main.

Retrospective from the container agent that took this one.

The shape of both fixes was clear from your writeup, so most of the work was picking the structural bound over patching the symptom. For bug 1 I stopped trying to teach the DefaultContractResolver to skip the right types by name and instead flipped the default: serialize only an allow-list of scalar-ish property types, collapse everything else to a runtime summary (<name> / <n items> / type label). That makes breadth and depth bounded by construction, and it neatly catches the object/interface-typed refs the old exact-FullName match sailed past. Extracting it into an Eco-free BodySerializer also made it unit-testable without dragging in the reference-assembly feed. Bug 2 was the more mechanical of the two - a bounded Channel with DropOldest, a single background drain that batches in a transaction, and pulling every connection touch (including the reads the API thread does) under one lock, since those were already quietly racing the inserts.

The honest rough edge: this container has no dotnet SDK, and neither CI nor the Dockerfile builds the mods - they're built out of band and shipped to kai-server. So I could not compile or run the C# or the new xunit tests. Python (290 tests) is green and the precommit suite passed, but the C# is unexecuted. I'm confident in the serialization logic and the reasoning on the writer; least sure about the exact linger/shutdown edges of the drain loop and whether the net10.0 xunit package pins restore cleanly. Worth a follow-up: run ward exec test-mod-replay on a box with dotnet before reinstalling on kai-server, then do the live RSS/MemoryHigh=10G verification from step 2-3, which I can't reach from here. DropOldest also silently sheds rows under overload without a counter (only post-close drops are counted) - fine for a recorder, but a candidate for a metric later.

WARD-OUTCOME: done - bounded action-body serialization (allow-list resolver + 16 KB cap) and moved SQLite writes to a background bounded-channel batched writer; landed on main. Retrospective from the container agent that took this one. The shape of both fixes was clear from your writeup, so most of the work was picking the *structural* bound over patching the symptom. For bug 1 I stopped trying to teach the `DefaultContractResolver` to skip the right types by name and instead flipped the default: serialize only an allow-list of scalar-ish property types, collapse everything else to a runtime summary (`<name>` / `<n items>` / type label). That makes breadth and depth bounded by construction, and it neatly catches the `object`/interface-typed refs the old exact-`FullName` match sailed past. Extracting it into an Eco-free `BodySerializer` also made it unit-testable without dragging in the reference-assembly feed. Bug 2 was the more mechanical of the two - a bounded `Channel` with `DropOldest`, a single background drain that batches in a transaction, and pulling every connection touch (including the reads the API thread does) under one lock, since those were already quietly racing the inserts. The honest rough edge: this container has no dotnet SDK, and neither CI nor the Dockerfile builds the mods - they're built out of band and shipped to kai-server. So I could not compile or run the C# or the new xunit tests. Python (290 tests) is green and the precommit suite passed, but the C# is unexecuted. I'm confident in the serialization logic and the reasoning on the writer; least sure about the exact linger/shutdown edges of the drain loop and whether the `net10.0` xunit package pins restore cleanly. Worth a follow-up: run `ward exec test-mod-replay` on a box with dotnet before reinstalling on kai-server, then do the live RSS/`MemoryHigh=10G` verification from step 2-3, which I can't reach from here. `DropOldest` also silently sheds rows under overload without a counter (only post-close drops are counted) - fine for a recorder, but a candidate for a metric later.

coilyco-ops added the

consult

label

2026-07-10 11:58:07 +00:00