Commit Graph

592 Commits

Author SHA1 Message Date
Micha 30aa696e61 Prepare Renovate bot against Gitea (F-12) + doc sweep
renovate.json: gitea platform, autodiscover Micha/*, group rules
(major separate, minor+patch+digest grouped, stateful tier-1
individual, komodo-major disabled), pin range strategy, no
automerge, dependency dashboard enabled.

ops/renovate/run-renovate.sh: one-shot docker run wrapper that
reads the Gitea PAT from /mnt/user/appdata/secrets/renovate_token.txt,
runs renovate/renovate:41, logs into /mnt/user/services/renovate/logs/.

docs/RENOVATE.md: 5-step operator setup (Gitea service account,
PAT, token file, first run, six-hourly user script). Explicit
no-automerge stance with notfall-stop checklist.

Cross-doc sweep: SECRETS_MAP entry for renovate_token.txt,
REPO_MAP entry for RENOVATE.md, AUDIT_2026-05-25_TODO new
Sprint 8 with F-15, F-07, F-09 rest, F-12 status, MIGRATION_LOG
captures the four-block sprint in one entry.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:29:20 +02:00
Micha e4b0db2af6 Add Komodo bootstrap dry-run scaffold (F-09 rest)
Mirror of the Immich restore-test pattern for the Komodo bootstrap
anchor. Brings up a throwaway komodo-mongo + komodo-core +
komodo-periphery under project restoretest-komodo, isolated from
production:

- same image digests as production (mongo:7.0.32, komodo-core:2,
  komodo-periphery:2) to prove compose-level bootstrap compatibility
- restore-lab paths under /mnt/user/backups/restore-lab/komodo
- 127.0.0.1:19120 only, no LAN bind, no Traefik, no Authelia
- test periphery runs WITHOUT docker.sock mount and WITHOUT
  /mnt/user/services mount; cannot manage productive containers
- KOMODO_* secrets are throwaway placeholders hardcoded in the test
  compose; productive secrets never enter this path

Smoke test: compose config valid, mongo healthy, mongo auth-ping
with test creds, komodo-core HTTP 200/302/303/401, periphery
container running. Report under restore-reports/komodo-bootstrap-*.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:25:41 +02:00
Micha 1a4929f9ef Pin monitoring stack images by digest
Reads live RepoDigests of each running monitoring container and
freezes the compose to the exact image manifest. Brings the
monitoring stack to the same digest-pin discipline as the
stateful tier-1 services. influxdb3-core was already pinned.

Affected: prometheus, alertmanager, alertmanager-ntfy-bridge,
blackbox-exporter, loki, promtail, grafana, node-exporter,
cadvisor (plus a second python:3.13-alpine for the bootstrap
dashboard importer).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:23:03 +02:00
Micha 2c0076c6a6 Fix vaultwarden + authelia healthcheck commands
Vaultwarden image ships curl, not wget. Switched the CMD-SHELL
test from wget --spider to curl -fsS.

Authelia 4.39.x removed the "helper health-check" subcommand;
use the /api/health endpoint via wget instead (verified inside
the running container).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:14:27 +02:00
Micha 7da64ff316 Add healthcheck to Authelia (authelia helper health-check)
Authelia ships its own health-check binary subcommand since 4.37+.
Avoids needing wget/curl in the container.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:51 +02:00
Micha 12b63531d1 Add healthcheck to Traefik (ping endpoint)
Enable --ping=true and use traefik healthcheck --ping. Lightweight
binary call inside the container, no extra tooling needed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:51 +02:00
Micha 3daea94982 Add healthcheck to Gitea (/api/healthz)
Gitea exposes /api/healthz unauthenticated. 60s start_period
because Gitea sqlite migration on cold start can take a while.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:51 +02:00
Micha 0ca29069c7 Add healthcheck to Vaultwarden (/alive)
Vaultwarden exposes /alive for liveness. wget --spider, 30s
interval, 30s start_period.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:50 +02:00
Micha eedb08316d Add healthcheck to Redis (redis-cli ping with auth)
Tier-1 health visibility for the shared Redis. Uses redis-cli with
the password from the mounted secret, fails on anything but PONG.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:50 +02:00
Micha 54a7a0e783 Add healthcheck to postgresql17 (pg_isready)
Tier-1 health visibility for shared Postgres cluster. pg_isready
against the admin DB; 30s interval, 30s start_period.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:09:50 +02:00
Micha c677ef0515 Add service removal checklist after stale Borg source finding
Befund vom 2026-05-29: HomelabBorgLastJobCompletedWithWarnings
zuendete vier Tage in Folge mit Borg-Exit-Code 107. Ursache im
Logfile: /local/appdata/homepage wurde am 25.05. entfernt, aber
in der Borg-UI-Source-Liste blieb der Eintrag drin und Borg
warnte taeglich BackupFileNotFoundError. Backups selbst waren
nicht gefaehrdet (alle 23 anderen Quellen sauber archiviert).

Operator hat den Eintrag in der Borg-UI manuell entfernt;
Source-Liste jetzt 23 statt 24, naechster Lauf 2026-05-30 sollte
wieder completed ohne Warning sein.

Erkenntnis: bei Stack-Removal wurde die Borg-Source-Liste nicht
mit-aufgeraeumt. WORKFLOW.md um neuen Abschnitt "Service-Removal-
Checkliste" erweitert mit 9 Pflichtschritten inklusive
Borg-UI-Source-Bereinigung als Schritt 8.

Positiv: die am 2026-05-27 scharfgeschaltete Alert-Pipeline
(Cron Textfile -> node-exporter -> Prometheus -> Alertmanager
-> ntfy-Bridge) hat den Drift binnen 24 h sichtbar gemacht.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-29 15:01:45 +02:00
Micha 2b60a58753 Activate H drive nearline pull as daily scheduled task
Windows Scheduled Task "KalliLab H Drive Nearline Pull" auf dem
Operator-Windows-PC registriert: taeglich 05:30 nach dem Borg-
Dump-Fenster. RunLevel Limited, StartWhenAvailable, Akku-OK,
Execution-Time-Limit 2h. Naechster Lauf 2026-05-29 05:30.

Repo-Snippet in H_DRIVE_NEARLINE_PULL.md korrigiert: PowerShell-
Enum-Wert ist Limited, nicht LeastPrivilege (alter Snippet haette
beim ersten Register-ScheduledTask einen Parameter-Binding-Fehler
geworfen). Status auf "produktiv" gesetzt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 20:25:26 +02:00
Micha 7d64248710 Decide against second offsite, keep paperless-gpt and BentoPDF
Operator-Entscheidungen 2026-05-28:

- F-03 zweites Off-site: bewusst NICHT umgesetzt. 3-2-1 ist mit
  Live + lokalem Borg + Hetzner + H:/-Nearline erfuellt; ein
  zweites Off-site deckt nur den Fall "Hetzner-Account verloren"
  ab, Aufwand unverhaeltnismaessig fuer Familien-Homelab.
  Stattdessen drei Folge-TODOs zur Haertung der bestehenden
  Topologie. Hetzner-2FA bewusst ohne (Operator-Praeferenz,
  analog USV-Risiko-Akzeptanz), durch starkes Passwort +
  Backup-Zahlungsweg + Login-Mails ersetzt. Borg-Append-Only-
  Befund: Repo laeuft im Mode 'full', custom_flags leer; Setup
  waere server-seitig in Hetzner-authorized_keys (Folge-Sprint).
  Review-Trigger in OFFSITE_BACKUP_OPTIONS.md dokumentiert.

- paperless-gpt: behalten bis Paperless-NGX 3.0 (erwartete
  native KI-Features). Aktuell 0 Traefik-Zugriffe in 7 Tagen,
  Resource-Footprint 34 MB RAM.

- BentoPDF: behalten als situatives Tool. 0 Traefik-Zugriffe,
  4 MB RAM. Begruendungs-Anker im SERVICE_CATALOG.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 20:19:53 +02:00
Micha edcb34c3f3 Record Plex reclaim and lock to LAN/Tailscale-only
Operator-Befund beim F-17-Versuch: Plex-Server war seit 18.05.
unclaimed (Preferences.xml ohne PlexOnline*) und Library-Sections
leer. Filmdateien unter /mnt/user/media/* blieben unangetastet.

Reclaim als Xeridos via inline PLEX_CLAIM-Env beim docker compose
force-recreate. Token nirgendwo persistiert (kein .env, kein Repo,
keine Komodo-Stack-ENV); zweiter Recreate ohne Token, damit
docker inspect-Snapshot sauber bleibt.

Endstand: PlexOnlineUsername Xeridos, PlexOnlineHome 1,
PublishServerOnPlexOnlineKey 0 (Remote Access aus). Bibliotheken
operator-seitig wieder eingerichtet (/data/movies 1.4 TB,
/data/Heimatfilme 300 GB). Plex bleibt LAN/Tailscale-only,
konsistent zur FRITZBox-Bereinigung vom selben Tag.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 20:06:36 +02:00
Micha 19604e0114 Record FRITZBox WAN cleanup (80/tcp out, VONETS UPnP off)
Operator umgesetzt 2026-05-28:
- 80/tcp aus FRITZBox-UI entfernt; Mobilfunk-validiert: http
  liefert Timeout, https weiter erreichbar.
- 222/tcp bleibt bewusst nicht eingerichtet (Tailscale-only-
  Linie). MASTER Sektion 10 entsprechend praezisiert.
- UPnP-Selbstfreigabe-Recht fuer PC-192-168-178-71 deaktiviert.
  Identifiziert als VONETS-WiFi-Bridge (vermutlich SolarEdge-
  Wechselrichter). SolarEdge-Cloud-Sync ist outbound und
  braucht keine UPnP.

Aktiver WAN-Endstand: ausschliesslich 443/tcp -> 192.168.178.58.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 15:32:10 +02:00
Micha 3c71a66c55 Document monitoring alerts, bundle cron and H/ pull live status
- AUDIT_2026-05-25_TODO: Borg-Stale, Cert-Expiry, Container-Down
  Alerts auf "erledigt" (Cron */5 textfile exporter live,
  Prometheus reload mit 14 Regeln); Gitea-Bundle-Cron auf "erledigt"
  (User-Script gitea-bundle-mirror-6h aktiv, Bundles 644);
  H:/ Nearline-Pull auf "erledigt (Pull live, Scheduled Task offen)"
  mit Zaehlerstaenden 19 Borg-Dumps + 10 Bundle-Files.

- MIGRATION_LOG: neuer Eintrag fasst die drei zusammenhaengenden
  Live-Aktivierungen zusammen, inkl. Befund-Ursprung (Permission-
  Drift), Reparaturen und expliziter Ausklammerung der nicht
  angefassten Themen (Auth, Hermes, USV, FRITZ!Box, Plex).

- H_DRIVE_NEARLINE_PULL: Erstlauf-Befund mit Permission-Issues
  und nachgezogenem Stand; Erwartungs-Liste auf real geliefertes
  Set angepasst; Flash-Config explizit Out-of-Scope.

- pull-critical-backups.ps1: Live-Robocopy-Output an Out-Null,
  damit der Markdown-Report nicht von Robocopy-Strings zerlegt
  wird (PowerShell-Pipeline-Quirk im foreach).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:48:04 +02:00
Micha 24d0d90670 Make dump output 0644 by default, exclude flash config from H pull
pre-backup-dumps.sh: atomic_write nimmt jetzt einen optionalen
mode-Parameter (Default 0644). Damit sind alle DB-/SQLite-/BoltDB-
/Mongo-Dumps konsistent 0644 und vom Nearline-Pull lesbar. Die
sensible unraid-flash-config-Familie (.tar.gz, .sha256, .manifest)
ruft explizit mit mode 600 auf und bleibt damit Operator-only.
Loest das Permission-Problem fuer filebrowser.bolt.dump (Source
ist 0640) im naechsten regulaeren Dump-Lauf.

pull-critical-backups.ps1: Jobs koennen ExcludeFiles ueber /XF
mitliefern. borg-dumps-latest schliesst die unraid-flash-config-
Artefakte aus, weil sie bewusst 0600 bleiben sollen und sonst den
Lauf abbrechen lassen. Restore-Quelle fuer Flash-Config bleibt
das Hetzner-Borg-Repo.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:44:50 +02:00
Micha 0ae44bd797 Write Prometheus textfile and Gitea bundles world-readable
node-exporter runs as nobody:65534 inside its container and was
hitting node_textfile_scrape_error 1 on homelab.prom, because the
file was 0600 root:root (mktemp default). Set it to 0644 right
before the atomic mv. Bundle inhaltsidentisch zum Git-Repo, ohne
Secrets (.gitignore-abgedeckt) und nicht sensibler als die
uebrigen /mnt/user/backups/borg/dumps/latest/*.dump-Files, die
ebenfalls 0644 sind. So funktioniert auch der Nearline-Pull-Workflow
ueber SMB (docs/H_DRIVE_NEARLINE_PULL.md).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:41:07 +02:00
Micha 0723eccca1 Sync repo map, audit TODO and migration log
Pull repo map up to include FAMILY_VIEW_DASHBOARD,
RESTORE_DRILL_ROUTINE, IMMICH_RESTORE_TEST, FRITZBOX_PORT_
CORRECTION_PLAN and OFFSITE_BACKUP_OPTIONS.

Mark Sprint 2 'Komodo bootstrap', Sprint 3 'Family-View',
Sprint 4 'Family onboarding' and Sprint 7 'Quarterly restore drill'
as done with explicit completion notes. Carry the FRITZBox UPnP
finding forward; tag the second offsite item as decision-pending.

Add two doc-only migration log entries for the bootstrap/family/
onboarding/drill sprint and for the FRITZBox/offsite preparation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:21:37 +02:00
Micha 3bfecdd291 Add FRITZBox correction plan and offsite options
Two operator decision documents, doc-only, no live action:

- docs/FRITZBOX_PORT_CORRECTION_PLAN.md prepares the three open
  router items: remove 80/tcp (no HTTP-01 in use), do not add
  222/tcp while Tailscale remains the operator path, deactivate
  the UPnP self-exposure from PC-192-168-178-71. Every step waits
  for operator go.

- docs/OFFSITE_BACKUP_OPTIONS.md compares rsync.net, BorgBase EU2
  and rotating cold disk for a second offsite target. Recommends
  rsync.net or cold disk; BorgBase EU2 is explicitly not
  recommended because it does not separate the provider risk.
  No provider booked, no costs triggered.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:21:03 +02:00
Micha c4fd4154db Document quarterly restore drill routine
New docs/RESTORE_DRILL_ROUTINE.md introduces a three-stage model:
weekly freshness check, monthly/bimonthly mini-restores, quarterly
DR sanity check. Tracks confirmed mini-restores (Vaultwarden, Gitea,
Paperless 2026-05-07; Immich 2026-05-27) and rotates services by
quarter Q1-Q4. Includes ten-point DR sanity check and abort rules
that point at the drift runbook. No host schedule is created; the
existing ops/restore-tests/schedule.md now references this routine
as the source for quarterly assignment.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:15:43 +02:00
Micha dddb33d900 Finalize family onboarding before invitation
Set status to final pre-invitation, soften the 2FA section to
app-specific 2FA (no SSO promise while Authelia-OIDC stays parked),
add a 'bewusst nicht versprochen' block (no single sign-on, no
24/7 SLA, no hotline support, no data sharing), and refine the
2FA loss guidance.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:06:38 +02:00
Micha 8eac93c1a5 Add Family-View dashboard specification
New docs/FAMILY_VIEW_DASHBOARD.md specifies the homelab-family-view
Grafana dashboard: 8 panels covering endpoints up, Borg freshness,
cert days, critical containers, disk usage, endpoint table, cert
table and container status. Includes PromQL queries, thresholds,
layout grid, datasource references, build order and smoke test.
Dashboard JSON is intentionally not created yet because the
Borg-stale / cert-expiry / container-down metrics from Sprint 3
are still pending.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:06:30 +02:00
Micha cfa02ce627 Document Komodo bootstrap in linear stages
Add explicit stages A-F to docs/SERVICES_RECOVERY.md: host/docker
baseline, repo source, secrets order, Komodo start, web/GitOps
validation, tier stack rollout. Recovery anchor is ops/komodo/
docker-compose.yml; the self-stack is explicitly not the anchor.
Link DISASTER_RECOVERY Phase 4 stage 3 to the new bootstrap section
and the stack-env-only secrets section in SECRETS_MAP.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:01:20 +02:00
Micha 52414c47be Record Immich restore test success 2026-05-27 18:38:14 +02:00
Micha a8c440d4da Read Immich v2 restore counts 2026-05-27 18:33:29 +02:00
Micha 12cf8fb728 Prepare Immich restore upload markers 2026-05-27 18:29:53 +02:00
Micha 5b0782a8fa Harden Immich restore smoke checks 2026-05-27 18:25:30 +02:00
Micha a805f03481 Retry Immich restore during Postgres startup 2026-05-27 18:18:55 +02:00
Micha 4feecf4a8e Make Immich restore database creation idempotent 2026-05-27 18:16:25 +02:00
Micha 2e84700326 Make Immich restore test create database 2026-05-27 18:14:40 +02:00
Micha 8a19c45485 Use Borg known_hosts in restore tests 2026-05-27 18:12:48 +02:00
Micha 6a445094bd Record FRITZBox port exposure drift 2026-05-27 18:06:43 +02:00
Micha fc59e35c57 Record alert metrics host smoke 2026-05-27 06:40:31 +02:00
Micha 8e111d1e04 Prepare monitoring alert rules 2026-05-27 06:38:57 +02:00
Micha 85a0eb4c3a Activate storage layout documentation 2026-05-27 06:31:03 +02:00
Micha 38c3d87722 Prepare H drive nearline pull 2026-05-27 06:25:47 +02:00
Micha c5d231a0db Prepare Immich restore smoke test 2026-05-26 21:33:01 +02:00
Micha 48099fb48d Update audit follow-up documentation 2026-05-26 20:24:50 +02:00
Micha 5c5ca2fcec Fix Gitea bundle mirror host run 2026-05-26 20:16:19 +02:00
Micha 3b438324dc Record UPS risk acceptance 2026-05-26 19:57:00 +02:00
Micha 0625594443 Record offline Borg passphrase backup 2026-05-26 19:53:08 +02:00
Micha 5936a4d9c1 Add Gitea bundle recovery script 2026-05-26 19:50:50 +02:00
Micha f77a69a0b2 Record audit baseline tag audit-2026-05-25-baseline 2026-05-26 19:44:50 +02:00
Micha f73cf48e41 Document external recovery dependencies 2026-05-26 19:44:14 +02:00
Micha eea2697ca1 Triage policy check warnings 2026-05-26 19:42:01 +02:00
Micha a3d77d7529 Document hardware capacity baseline 2026-05-26 19:39:42 +02:00
Micha 02a50e1a58 Record Komodo metadata fix 2026-05-26 19:32:26 +02:00
Micha 267e76059a Clean up Komodo webhook drift 2026-05-26 19:29:51 +02:00
Micha 9d4fee02ca Record next audit handoff 2026-05-26 15:39:41 +02:00