Commit Graph

571 Commits

Author SHA1 Message Date
Micha e9a7f79025 Clarify audit doc-only deployment state 2026-05-25 07:36:03 +02:00
Micha 43727151df Refresh final audit live status 2026-05-25 07:34:08 +02:00
Micha 66ee10cb55 Clarify Disk1 parity follow-up 2026-05-25 06:17:13 +02:00
Micha ab68900216 Complete Disk1 phase 2 migration 2026-05-25 06:13:50 +02:00
Micha 8f56c6edcd Document Disk1 phase 2 backup readiness 2026-05-24 13:07:45 +02:00
Micha 8e400fb3c3 Finalize homelab audit end state 2026-05-23 11:29:08 +02:00
Micha cd650b19ac Close Gitea signup, dedup posture-check alerts, extend Borg scope
Operational hardening across several services after live incident
analysis between 2026-05-18 and 2026-05-20:

- Gitea: disable public registration and OpenID signup/signin to
  stop the external POST / 5xx bursts that triggered availability
  alerts. New repo-wide policy requires every productive
  Micha/homelab-infra Komodo stack to ship with an active
  Gitea->Komodo webhook on the current stack ID (documented in
  CLAUDE.md, AI_CONTEXT.md, WORKFLOW.md).
- posture-check: extract the Disk1 fstype check into its own
  function so the documented Disk1 NTFS exception no longer raises
  ntfy warnings, skip POSIX inode checks on NTFS, and dedup ntfy
  alerts via a fingerprint state file with ALERT_REPEAT_SECONDS
  (default 24h). Repeat-spam on the same cause now suppressed.
- docker-critical-events: parse the event JSON for container name,
  action, exit code and signal; drop `die exit=0` events (clean
  stops); ship a structured ntfy message instead of the raw event
  line.
- Borg UI: mount /mnt/user/services into the backup container as
  /local/services:ro and include homelab-infra, stacks and
  posture-check in all-important-sources.txt. RESTORE_MATRIX and
  DISASTER_RECOVERY updated accordingly.
- Unraid user scripts: document the new
  homelab-operations-report-daily cron job and the SMTP password
  file it expects on the host.
- MIGRATION_LOG: capture the four live events from this window -
  Gitea 5xx burst + signup closure, Komodo webhook reconciliation,
  posture-check host-version verification, Borg scope extension,
  and Traefik 5xx alert detuning.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:05:35 +02:00
Codex af231dd4e8 Fix zero-count noise pattern handling 2026-05-23 11:03:02 +02:00
Micha 428223d2e4 Mark posture report scripts executable 2026-05-23 11:00:40 +02:00
Micha b6d3ed4832 Tune homelab availability alerts 2026-05-23 10:58:12 +02:00
Micha 9e7bebbd3c Add daily operations report with hardened log-noise filtering
Brings the previously untracked daily-status-report.sh and
send-operations-report-mail.sh into the repo, plus a refactor of the
log-noise pipeline:

- New helper services/posture-check/lib/normalize-noise-patterns.sh
  strips comments, empty lines and trailing whitespace from
  log-noise.patterns before grep -f sees it. A stray empty line in
  the pattern file would otherwise have made grep -Eaif match every
  hit and silently wipe the log highlights.
- log-noise.patterns is now documented per-pattern (Why / Re-check).
  The Vaultwarden pattern is split: token/session noise stays as
  noise; DNS/Connect/Resolve/reqwest/hyper errors are removed from
  the noise set so real network signals stay visible.
- collect_log_highlights now reports a per-container and per-pattern
  noise breakdown (Top N) and an escalation flag when any pattern
  exceeds NOISE_ESCALATION_THRESHOLD (default 500). The flag is fed
  into derive_report_status and the management summary.
- New shell tests under services/posture-check/tests/ verify the
  normalize helper handles comments, empty lines, whitespace-only
  lines, and that unknown error lines remain in the attention set.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 10:41:33 +02:00
Micha b7cbbe51de Fix Jellyfin external DNS 2026-05-18 20:29:18 +02:00
Micha 71ac18b21c Fix Jellyfin native auth routing 2026-05-18 13:43:41 +02:00
Micha 90f270be96 Fix Jellyfin config permissions 2026-05-18 13:21:30 +02:00
Micha e28f8dabec Add Jellyfin media server stack 2026-05-18 13:09:32 +02:00
Micha edfec5b66d Add Plex media server stack 2026-05-18 13:09:27 +02:00
Micha 59bec9ac77 Fix Glance live widget data sources 2026-05-18 09:35:53 +02:00
Micha 9f86da708a Add Glance live network widgets 2026-05-18 08:31:57 +02:00
Micha d6170211c4 Refine Glance network widgets 2026-05-18 08:13:13 +02:00
Micha fb681086f3 Restyle Glance dashboard layout 2026-05-18 08:03:59 +02:00
Micha 5b101f3b3d Keep only verified Glance community widget 2026-05-17 18:20:56 +02:00
Micha 669efbd57e Fix Glance Speedtest subrequest headers 2026-05-17 18:18:14 +02:00
Micha 2dd5590a2a Polish Glance community widgets 2026-05-17 18:16:05 +02:00
Micha 175cd6951f Add Glance community homelab widgets 2026-05-17 18:07:57 +02:00
Micha aeb7573b03 Remove noisy Glance dashboard widgets 2026-05-17 17:08:10 +02:00
Micha 215f44b962 Fix Glance monitor health checks 2026-05-17 17:05:23 +02:00
Micha 6ce625f77a Fix Glance socket proxy image tag 2026-05-17 16:59:29 +02:00
Micha c3c8060ddf Add Glance homelab dashboard stack 2026-05-17 16:51:43 +02:00
Micha 29eaf8001f Normalize ntfy alert routing 2026-05-17 14:57:45 +02:00
Micha db7dc3f2af Add ntfy alert delivery for monitoring 2026-05-17 11:34:19 +02:00
Micha c748236886 Prune monitoring dashboard imports 2026-05-17 11:30:00 +02:00
Micha 8aa850df40 Set Grafana DNS resolvers 2026-05-17 11:26:27 +02:00
Micha 2c4854f628 Accept protected HTTP checks in blackbox 2026-05-17 11:25:35 +02:00
Micha b7050812d4 Fix blackbox DNS resolution 2026-05-17 11:24:20 +02:00
Micha c95fa601f0 Add monitoring replacement baseline 2026-05-17 11:22:38 +02:00
Micha 0c308ff352 Preserve InfluxDB data in monitoring stack 2026-05-17 10:47:57 +02:00
Micha 53216e50c1 Fix monitoring InfluxDB volume permissions 2026-05-17 10:45:32 +02:00
Micha b7dfdad621 Consolidate monitoring target stack 2026-05-17 10:41:29 +02:00
Micha 61625a7a1c ops: keep monitoring importer running for komodo 2026-05-16 22:39:09 +02:00
Micha 6e28ea94d2 ops: wire monitoring stack to traefik metrics 2026-05-16 22:10:43 +02:00
Micha 58eb53a6a8 ops: add monitoring compose stack 2026-05-16 21:59:20 +02:00
Micha d345d770c2 docs: add homelab audit report 2026-05-16 21:51:48 +02:00
Micha 2e136d9060 Update current homelab rest list 2026-05-16 20:30:31 +02:00
Micha 6ca829ec45 Document Unraid automation schedules 2026-05-16 20:11:19 +02:00
Micha ef3b546d30 Align documentation consistency fixes 2026-05-16 20:04:46 +02:00
Micha 6f684fb4e3 Document Unraid native log rotation 2026-05-16 19:31:54 +02:00
Micha 0adddb6533 Add Unraid automation script templates 2026-05-16 14:34:35 +02:00
Micha 162421e537 Harden Gitea webhooks and Docker log limits 2026-05-16 13:34:45 +02:00
Micha bf30240217 Remove Loki image-internal healthcheck 2026-05-16 13:32:02 +02:00
Micha 5f7940aa01 Tune Loki host bootstrap settings 2026-05-16 13:31:08 +02:00