coilyco-flight-deck/infrastructure

Fork 0

Self-hosting evaluation backlog (from selfh.st/apps) #326

New issue

Closed

opened 2026-06-15 00:08:07 +00:00 by coilysiren · 3 comments

coilysiren commented

2026-06-15 00:08:07 +00:00

Owner

Self-hosting evaluation backlog sourced from selfh.st/apps, filtered against the platform-engineer / kai-server stack. Each line is a candidate to evaluate and either deploy on kai-server or reject with a reason. Star counts are a popularity signal at time of filing (2026-06-14).

Secrets / Identity

OpenBao (6.3k) - open Vault fork, secrets/certs/keys. Evaluate as the IaC-native secrets store alongside (or behind) the existing SSM-everything discipline. Compare to Infisical (27k) as the lighter UI-first alternative.

Monitoring / Dashboard

Beszel vs Glance - decide the kai-server "home base" surface. Beszel (23k) = lightweight server-monitoring hub; Glance (35k) = dashboard centralizing feeds/status/widgets. Possibly both (monitoring vs feed dashboard) rather than either/or.
Uptime-Kuma (88k) - simple self-hosted uptime monitor + status page. Compare against Gatus (developer-oriented, config-as-code) before committing.

Fleet / Deployment

Komodo (11k) - build/deploy across multiple servers, Docker-native. Terraform-adjacent fleet management for kai-server + tower.
Semaphore UI (14k) - web UI for Ansible / Terraform / OpenTofu runs. Fits the authored-here / ansible-rollout split; evaluate as a front end for existing infrastructure/ansible roles. Note: orchestrates/triggers runs - distinct layer from ARA below, which records/reports them.
ARA (records.ansible) - ansible callback plugin that records every run (task/host/result) to a DB, incl. --check --diff drift runs. Backend (on Postgres) for the NocoDB drift-grid pipeline - see comment. Read/reporting layer, distinct from Semaphore's execute layer.
Scrutiny (7.9k) - drive S.M.A.R.T. monitoring across fleet hardware.

Automation / Internal tooling

n8n (192k) - workflow automation. Could absorb some Trello/Discord/Forgejo glue currently hand-rolled.
Windmill (17k) - Python-first developer infra for internal tools/scripts with generated UIs. Compare to n8n: Windmill is script-centric, n8n is node-centric.
NocoDB (63k) - Postgres-backed Airtable-ish no-code DB/spreadsheet for internal tooling. Compare to Teable (21k) / Grist (11k) if NocoDB falls short. First concrete use case: Airtable-ish grid over ARA's Postgres (external-base connection, no ETL) for the backfill/rollout drift matrix - see comment.

AI / LLM

Manifest (7k) - smart model router for personal AI agents. Directly in the OpenClaw / LLM-API lane.
Open WebUI (141k) - extensible local-LLM / multi-provider chat front end. The standard interface; evaluate for day-to-day LLM-API testing.

Notes

Memos (61k) - fast instant-capture note stream. Evaluate as a quick-capture inbox that complements (does not replace) the Obsidian vault.

Categories to pick a tool within (research + choose, then split into its own task)

Site analytics tool - privacy-friendly web analytics for coilysiren.me surfaces. Candidates: Umami (37k), Plausible (27k), Matomo (21k), Rybbit (12k), GoatCounter (5.8k). PostHog (35k) if product-analytics depth is wanted over simple pageviews.
Error tracking: Sentry vs Bugsnag - Note: Bugsnag is not meaningfully self-hostable (commercial SaaS). The real self-host shortlist is Sentry (44k, already used via MCP), GlitchTip (lightweight Sentry-compatible), or Bugsink (1.9k). Decide: self-host full Sentry vs run GlitchTip/Bugsink against the same SDKs.
RSS - feed reader / aggregator. Candidates: FreshRSS (15k), Miniflux (9.4k, minimalist), CommaFeed (3.5k). Pair with RSSHub (44k) to generate feeds from sources that lack them. (Note: Glance above can also surface feeds, so scope this against that decision.)

Source: selfh.st/apps CDN catalog (1,297 apps), filtered 2026-06-14. Generated via Claude Code.

Self-hosting evaluation backlog sourced from [selfh.st/apps](https://selfh.st/apps/), filtered against the platform-engineer / kai-server stack. Each line is a candidate to evaluate and either deploy on kai-server or reject with a reason. Star counts are a popularity signal at time of filing (2026-06-14). ## Secrets / Identity - [ ] **OpenBao** (6.3k) - open Vault fork, secrets/certs/keys. Evaluate as the IaC-native secrets store alongside (or behind) the existing SSM-everything discipline. Compare to Infisical (27k) as the lighter UI-first alternative. ## Monitoring / Dashboard - [ ] **Beszel vs Glance** - decide the kai-server "home base" surface. Beszel (23k) = lightweight server-monitoring hub; Glance (35k) = dashboard centralizing feeds/status/widgets. Possibly both (monitoring vs feed dashboard) rather than either/or. - [ ] **Uptime-Kuma** (88k) - simple self-hosted uptime monitor + status page. Compare against Gatus (developer-oriented, config-as-code) before committing. ## Fleet / Deployment - [ ] **Komodo** (11k) - build/deploy across multiple servers, Docker-native. Terraform-adjacent fleet management for kai-server + tower. - [ ] **Semaphore UI** (14k) - web UI for Ansible / Terraform / OpenTofu runs. Fits the authored-here / ansible-rollout split; evaluate as a front end for existing infrastructure/ansible roles. Note: *orchestrates/triggers* runs - distinct layer from ARA below, which *records/reports* them. - [ ] **ARA** (records.ansible) - ansible callback plugin that records every run (task/host/result) to a DB, incl. `--check --diff` drift runs. Backend (on Postgres) for the NocoDB drift-grid pipeline - see comment. Read/reporting layer, distinct from Semaphore's execute layer. - [ ] **Scrutiny** (7.9k) - drive S.M.A.R.T. monitoring across fleet hardware. ## Automation / Internal tooling - [ ] **n8n** (192k) - workflow automation. Could absorb some Trello/Discord/Forgejo glue currently hand-rolled. - [ ] **Windmill** (17k) - Python-first developer infra for internal tools/scripts with generated UIs. Compare to n8n: Windmill is script-centric, n8n is node-centric. - [ ] **NocoDB** (63k) - Postgres-backed Airtable-ish no-code DB/spreadsheet for internal tooling. Compare to Teable (21k) / Grist (11k) if NocoDB falls short. First concrete use case: Airtable-ish grid over ARA's Postgres (external-base connection, no ETL) for the backfill/rollout drift matrix - see comment. ## AI / LLM - [ ] **Manifest** (7k) - smart model router for personal AI agents. Directly in the OpenClaw / LLM-API lane. - [ ] **Open WebUI** (141k) - extensible local-LLM / multi-provider chat front end. The standard interface; evaluate for day-to-day LLM-API testing. ## Notes - [ ] **Memos** (61k) - fast instant-capture note stream. Evaluate as a quick-capture inbox that complements (does not replace) the Obsidian vault. ## Categories to pick a tool within (research + choose, then split into its own task) - [ ] **Site analytics tool** - privacy-friendly web analytics for coilysiren.me surfaces. Candidates: Umami (37k), Plausible (27k), Matomo (21k), Rybbit (12k), GoatCounter (5.8k). PostHog (35k) if product-analytics depth is wanted over simple pageviews. - [ ] **Error tracking: Sentry vs Bugsnag** - Note: Bugsnag is not meaningfully self-hostable (commercial SaaS). The real self-host shortlist is Sentry (44k, already used via MCP), GlitchTip (lightweight Sentry-compatible), or Bugsink (1.9k). Decide: self-host full Sentry vs run GlitchTip/Bugsink against the same SDKs. - [ ] **RSS** - feed reader / aggregator. Candidates: FreshRSS (15k), Miniflux (9.4k, minimalist), CommaFeed (3.5k). Pair with RSSHub (44k) to generate feeds from sources that lack them. (Note: Glance above can also surface feeds, so scope this against that decision.) --- Source: selfh.st/apps CDN catalog (1,297 apps), filtered 2026-06-14. Generated via Claude Code.

coilysiren referenced this issue

2026-06-15 00:18:06 +00:00

Introduce CloudNativePG as shared app Postgres; Umami as first tenant #327

coilysiren referenced this issue

2026-06-15 00:18:07 +00:00

Migrate forgejo's in-pod Postgres onto CloudNativePG (later) #328

coilysiren commented

2026-06-15 00:18:27 +00:00

Author

Owner

Decisions (2026-06-15)

Site analytics → Umami. Picked over Plausible (Plausible needs both Postgres and ClickHouse; Umami is single-Postgres) and the heavier PostHog/Matomo. Deployment is now tracked under the shared-Postgres work, since Umami is the cluster's second Postgres-needing app:

#327 (CloudNativePG + Umami as first tenant)
#328 (later: migrate forgejo's in-pod Postgres onto it)

Error tracking → stay on Sentry cloud; drop the self-host-Sentry idea. Self-hosting real Sentry is a ~40-container, 16GB+ stack - not worth it on a homelab k3s node. Kai already uses Sentry SaaS and likes it, so the least-ops answer is to keep it. GlitchTip (Sentry-SDK-compatible, ~1-2GB) remains a fallback only if data-local self-hosting ever becomes a goal; the existing docs/glitchtip-deploy-plan.md is parked, not active.

## Decisions (2026-06-15) **Site analytics → Umami.** Picked over Plausible (Plausible needs both Postgres *and* ClickHouse; Umami is single-Postgres) and the heavier PostHog/Matomo. Deployment is now tracked under the shared-Postgres work, since Umami is the cluster's second Postgres-needing app: - https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/327 (CloudNativePG + Umami as first tenant) - https://forgejo.coilysiren.me/coilyco-flight-deck/infrastructure/issues/328 (later: migrate forgejo's in-pod Postgres onto it) **Error tracking → stay on Sentry cloud; drop the self-host-Sentry idea.** Self-hosting real Sentry is a ~40-container, 16GB+ stack - not worth it on a homelab k3s node. Kai already uses Sentry SaaS and likes it, so the least-ops answer is to keep it. GlitchTip (Sentry-SDK-compatible, ~1-2GB) remains a fallback *only* if data-local self-hosting ever becomes a goal; the existing `docs/glitchtip-deploy-plan.md` is parked, not active.

coilysiren commented

2026-06-15 13:12:36 +00:00

Author

Owner

ARA → Postgres → NocoDB drift-grid pipeline

Concrete path for using ARA as the ansible run recorder and NocoDB as an Airtable-ish grid over it, for tracking the backfill/grandfather/rollout matrix. The trick is NocoDB's "connect to an existing database as an external source" feature, which lets us skip ETL entirely:

ARA backend → PostgreSQL (ARA supports Postgres, not just SQLite). ARA is itself an ansible callback plugin, so it records every task/host/result of every run, including --check --diff drift runs.
Postgres view collapsing ARA's run-history into current-state-per-(task, host) - latest changed/ok per backfill per host. That view is the drift matrix.
NocoDB → that same Postgres as an external base, exposing the view as a grid (backfill × host, color by status).

No copy, no sync job, no cron shuffling records - NocoDB reads ARA's live tables and the matrix collapse is just SQL. The glue tools elsewhere on this issue (n8n/Windmill) are only needed if we want to push back (e.g. open/close Forgejo issues from the grid), not for the read view.

Notes / decisions this implies

ARA vs Semaphore UI are different layers: ARA records and reports (read), Semaphore orchestrates and triggers runs (execute). For the drift-grid use case we only need ARA + NocoDB; Semaphore is orthogonal. Decide the boundary before deploying both.
Dependency order: ARA recording → Postgres backend → NocoDB external base. The pipeline isn't a single deploy; it's a downstream consequence of three line items here. ARA wasn't an explicit item, so I've added it under Fleet/Deployment.
Authored-here / ansible-rollout split still holds: the drift check + ARA wiring is authored in infrastructure/ansible; an ansible role does the kai-server rollout of ARA/NocoDB.
Source of truth stays the ansible check. The grid is a materialized view of it, not a second backlog to hand-maintain.

## ARA → Postgres → NocoDB drift-grid pipeline Concrete path for using **ARA** as the ansible run recorder and **NocoDB** as an Airtable-ish grid over it, for tracking the backfill/grandfather/rollout matrix. The trick is NocoDB's "connect to an existing database as an external source" feature, which lets us skip ETL entirely: 1. **ARA backend → PostgreSQL** (ARA supports Postgres, not just SQLite). ARA is itself an ansible callback plugin, so it records every task/host/result of every run, including `--check --diff` drift runs. 2. **Postgres view** collapsing ARA's run-history into current-state-per-`(task, host)` - latest `changed`/`ok` per backfill per host. That view *is* the drift matrix. 3. **NocoDB → that same Postgres as an external base**, exposing the view as a grid (backfill × host, color by status). No copy, no sync job, no cron shuffling records - NocoDB reads ARA's live tables and the matrix collapse is just SQL. The glue tools elsewhere on this issue (n8n/Windmill) are only needed if we want to *push back* (e.g. open/close Forgejo issues from the grid), not for the read view. ### Notes / decisions this implies - **ARA vs Semaphore UI** are different layers: ARA *records and reports* (read), Semaphore *orchestrates and triggers* runs (execute). For the drift-grid use case we only need ARA + NocoDB; Semaphore is orthogonal. Decide the boundary before deploying both. - **Dependency order**: ARA recording → Postgres backend → NocoDB external base. The pipeline isn't a single deploy; it's a downstream consequence of three line items here. ARA wasn't an explicit item, so I've added it under Fleet/Deployment. - Authored-here / ansible-rollout split still holds: the drift check + ARA wiring is authored in `infrastructure/ansible`; an ansible role does the kai-server rollout of ARA/NocoDB. - Source of truth stays the ansible check. The grid is a materialized view of it, not a second backlog to hand-maintain.

coilysiren referenced this issue

2026-06-15 13:13:21 +00:00

Evaluate ARA + NocoDB drift-grid pipeline (first self-host eval) #330

coilysiren added the

label

2026-06-17 07:07:09 +00:00

coilysiren commented

2026-06-17 08:22:31 +00:00

Author

Owner

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: burndown-2026-06.

Backlog burndown 2026-06-17: closing low-priority (P3/P4) to bring the open count to a manageable level. Nothing lost — reopen if this resurfaces. Batch tag: `burndown-2026-06`.

coilysiren

2026-06-17 08:22:31 +00:00