18 Commits

Author SHA1 Message Date
Micha 16a266cd79 Add GitOps runtime image drift alert 2026-05-31 13:17:45 +02:00
Micha 8095ab8b5d F-10: automated Authelia repo<->host drift check
New services/authelia-diff.sh compares the access_control: section of the
repo baseline against the live host configuration.yml. OIDC clients,
identity providers, and secret values stay out of scope by design.
Exit codes: 0 ok, 1 drift, 2 file missing, 3 section missing, 4 tool missing.

posture-check.sh gains check_authelia_config_drift, which calls the diff
script and reports drift as warning (not critical). SKIP_AUTHELIA_DRIFT=1
opts out; AUTHELIA_DIFF_SCRIPT overrides the path.

WORKFLOW.md gets a dedicated "Ausnahme: Authelia configuration.yml" section
analogous to the Traefik dynamic-config exception, with the mandatory
repo->host merge workflow and the env-variable contract.

Smoke-tested locally: identical files rc=0, ACL change rc=1 with proper
unified diff, non-ACL change (session.default_redirection_url) correctly
ignored.

Operator follow-up: set up a read-only repo mirror at
/mnt/user/services/homelab-infra/ so the check finds a current baseline.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-30 09:52:16 +02:00
Micha 0ae44bd797 Write Prometheus textfile and Gitea bundles world-readable
node-exporter runs as nobody:65534 inside its container and was
hitting node_textfile_scrape_error 1 on homelab.prom, because the
file was 0600 root:root (mktemp default). Set it to 0644 right
before the atomic mv. Bundle inhaltsidentisch zum Git-Repo, ohne
Secrets (.gitignore-abgedeckt) und nicht sensibler als die
uebrigen /mnt/user/backups/borg/dumps/latest/*.dump-Files, die
ebenfalls 0644 sind. So funktioniert auch der Nearline-Pull-Workflow
ueber SMB (docs/H_DRIVE_NEARLINE_PULL.md).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:41:07 +02:00
Micha 8e111d1e04 Prepare monitoring alert rules 2026-05-27 06:38:57 +02:00
Micha b6bbca43ad Replace Uptime Kuma with monitoring checks 2026-05-25 16:37:46 +02:00
Micha a7797fd02e Consolidate dashboard on Glance 2026-05-25 14:44:46 +02:00
Micha ab68900216 Complete Disk1 phase 2 migration 2026-05-25 06:13:50 +02:00
Micha cd650b19ac Close Gitea signup, dedup posture-check alerts, extend Borg scope
Operational hardening across several services after live incident
analysis between 2026-05-18 and 2026-05-20:

- Gitea: disable public registration and OpenID signup/signin to
  stop the external POST / 5xx bursts that triggered availability
  alerts. New repo-wide policy requires every productive
  Micha/homelab-infra Komodo stack to ship with an active
  Gitea->Komodo webhook on the current stack ID (documented in
  CLAUDE.md, AI_CONTEXT.md, WORKFLOW.md).
- posture-check: extract the Disk1 fstype check into its own
  function so the documented Disk1 NTFS exception no longer raises
  ntfy warnings, skip POSIX inode checks on NTFS, and dedup ntfy
  alerts via a fingerprint state file with ALERT_REPEAT_SECONDS
  (default 24h). Repeat-spam on the same cause now suppressed.
- docker-critical-events: parse the event JSON for container name,
  action, exit code and signal; drop `die exit=0` events (clean
  stops); ship a structured ntfy message instead of the raw event
  line.
- Borg UI: mount /mnt/user/services into the backup container as
  /local/services:ro and include homelab-infra, stacks and
  posture-check in all-important-sources.txt. RESTORE_MATRIX and
  DISASTER_RECOVERY updated accordingly.
- Unraid user scripts: document the new
  homelab-operations-report-daily cron job and the SMTP password
  file it expects on the host.
- MIGRATION_LOG: capture the four live events from this window -
  Gitea 5xx burst + signup closure, Komodo webhook reconciliation,
  posture-check host-version verification, Borg scope extension,
  and Traefik 5xx alert detuning.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 11:05:35 +02:00
Codex af231dd4e8 Fix zero-count noise pattern handling 2026-05-23 11:03:02 +02:00
Micha 428223d2e4 Mark posture report scripts executable 2026-05-23 11:00:40 +02:00
Micha 9e7bebbd3c Add daily operations report with hardened log-noise filtering
Brings the previously untracked daily-status-report.sh and
send-operations-report-mail.sh into the repo, plus a refactor of the
log-noise pipeline:

- New helper services/posture-check/lib/normalize-noise-patterns.sh
  strips comments, empty lines and trailing whitespace from
  log-noise.patterns before grep -f sees it. A stray empty line in
  the pattern file would otherwise have made grep -Eaif match every
  hit and silently wipe the log highlights.
- log-noise.patterns is now documented per-pattern (Why / Re-check).
  The Vaultwarden pattern is split: token/session noise stays as
  noise; DNS/Connect/Resolve/reqwest/hyper errors are removed from
  the noise set so real network signals stay visible.
- collect_log_highlights now reports a per-container and per-pattern
  noise breakdown (Top N) and an escalation flag when any pattern
  exceeds NOISE_ESCALATION_THRESHOLD (default 500). The flag is fed
  into derive_report_status and the management summary.
- New shell tests under services/posture-check/tests/ verify the
  normalize helper handles comments, empty lines, whitespace-only
  lines, and that unknown error lines remain in the attention set.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 10:41:33 +02:00
Micha 29eaf8001f Normalize ntfy alert routing 2026-05-17 14:57:45 +02:00
Micha 6ca829ec45 Document Unraid automation schedules 2026-05-16 20:11:19 +02:00
Micha 6f684fb4e3 Document Unraid native log rotation 2026-05-16 19:31:54 +02:00
Micha 0adddb6533 Add Unraid automation script templates 2026-05-16 14:34:35 +02:00
Micha a5add937f8 Add Loki Alloy logging baseline 2026-05-16 13:26:49 +02:00
Micha 23262cd7b9 Allow Disk1 NTFS posture transition 2026-05-16 13:12:19 +02:00
Micha 878ad2d5f1 Harden backup and posture checks 2026-05-16 13:04:22 +02:00