Add Unraid flash config to Borg preflight

This commit is contained in:
2026-05-25 19:36:16 +02:00
parent 09eeac51e1
commit d50b11784d
10 changed files with 88 additions and 9 deletions
+6 -2
View File
@@ -61,7 +61,7 @@ Diese Punkte sollten **vor** einem echten Ausfall geklaert sein:
| Thema | Sollzustand | | Thema | Sollzustand |
|---|---| |---|---|
| Repo-Zugang ausserhalb von Gitea | privater GitHub-Push-Mirror `michaelkaleschke-spec/homelab-infra` und lokaler aktueller Clone vorhanden | | Repo-Zugang ausserhalb von Gitea | privater GitHub-Push-Mirror `michaelkaleschke-spec/homelab-infra` und lokaler aktueller Clone vorhanden |
| Unraid USB-/Flash-Backup | eingerichtet und wiederherstellbar | | Unraid USB-/Flash-Backup | `unraid-flash-config.tar.gz` wird vor Borg unter `/mnt/user/backups/borg/dumps/latest` erzeugt und nach Hetzner/Borg gesichert; Unraid-Connect-Cloud-Backup optional zusaetzlich |
| Borg-Ziel | nicht nur lokal auf demselben Ausfallpfad | | Borg-Ziel | nicht nur lokal auf demselben Ausfallpfad |
| Borg-Passphrase | Host-Secret-Datei vorhanden und fuer Borg-Zugriff verifiziert; externe analoge Hinterlegung bleibt Operator-Aufgabe | | Borg-Passphrase | Host-Secret-Datei vorhanden und fuer Borg-Zugriff verifiziert; externe analoge Hinterlegung bleibt Operator-Aufgabe |
| Secrets-Dateien | ueber Borg bzw. Restore-Quellen abgedeckt | | Secrets-Dateien | ueber Borg bzw. Restore-Quellen abgedeckt |
@@ -100,6 +100,8 @@ Wenn **weder GitHub-Mirror noch lokaler Repo-Clone** verfuegbar sind, ist `servi
3. Array wieder zuweisen und starten 3. Array wieder zuweisen und starten
4. Grundlegende Shares pruefen 4. Grundlegende Shares pruefen
Primaere lokale/off-site Restore-Quelle fuer die bestehende Flash-Konfiguration ist das Borg-Artefakt `unraid-flash-config.tar.gz` aus `/mnt/user/backups/borg/dumps/latest`. Dieses Archiv enthaelt `/boot/config` und muss wie Secret-Material behandelt werden.
### 5.2 Erwartete Shares / Pfade ### 5.2 Erwartete Shares / Pfade
Mindestens diese Pfade muessen wieder verfuegbar sein: Mindestens diese Pfade muessen wieder verfuegbar sein:
@@ -207,6 +209,7 @@ Besonders kritisch:
- `/mnt/user/appdata/komodo/core` - `/mnt/user/appdata/komodo/core`
- `/mnt/user/appdata/komodo/periphery` - `/mnt/user/appdata/komodo/periphery`
- `/mnt/user/backups/borg/dumps/latest` - `/mnt/user/backups/borg/dumps/latest`
- `/mnt/user/backups/borg/dumps/latest/unraid-flash-config.tar.gz`
- dienstspezifische App- und Nutzdatenpfade - dienstspezifische App- und Nutzdatenpfade
**Nicht blind alles extrahieren**, wenn nur einzelne Pfade oder Dienste betroffen sind. **Nicht blind alles extrahieren**, wenn nur einzelne Pfade oder Dienste betroffen sind.
@@ -368,6 +371,7 @@ Relevant:
- Dump-Ziel: `/mnt/user/backups/borg/dumps/latest` - Dump-Ziel: `/mnt/user/backups/borg/dumps/latest`
- Skript: `ops/borg-ui/scripts/pre-backup-dumps.sh` - Skript: `ops/borg-ui/scripts/pre-backup-dumps.sh`
- Unraid-Flash-Artefakt: `unraid-flash-config.tar.gz` plus `.sha256` und Manifest im selben Zielpfad
### Hermes Agent ### Hermes Agent
@@ -390,7 +394,7 @@ Smoke-Test: `hermes-gateway` healthcheck ist gruen, `hermes.kaleschke.info` leit
## 11. Offene Vorbereitungs-To-dos ## 11. Offene Vorbereitungs-To-dos
- Unraid-USB-/Flash-Backup pruefen - Unraid-USB-/Flash-Backup regelmaessig ueber `unraid-flash-config.tar.gz` und optional Unraid Connect pruefen
- Borg-Passphrase aus `/mnt/user/appdata/secrets/borg_repo_passphrase.txt` extern analog sicher hinterlegen - Borg-Passphrase aus `/mnt/user/appdata/secrets/borg_repo_passphrase.txt` extern analog sicher hinterlegen
- Komodo Stack-ENV-Werte zentral ausserhalb von Komodo dokumentieren - Komodo Stack-ENV-Werte zentral ausserhalb von Komodo dokumentieren
- regelmaessige automatisierte Restore-Smoke-Tests fuer Vaultwarden, Gitea und Paperless etablieren - regelmaessige automatisierte Restore-Smoke-Tests fuer Vaultwarden, Gitea und Paperless etablieren
+5
View File
@@ -16,6 +16,11 @@ Dieses Dokument ist nur noch ein historischer Verlauf. Der aktuelle operative Ab
## Historische Meilensteine ## Historische Meilensteine
### 2026-05-25 - Unraid Flash-Backup in Borg-Scope aufgenommen
- `pre-backup-dumps.sh` erzeugt zusaetzlich zu den DB-Dumps ein sensibles `unraid-flash-config.tar.gz` aus `/boot/config` inklusive SHA256 und Manifest unter `/mnt/user/backups/borg/dumps/latest`.
- Da `/local/borg-dumps` bereits Teil des Borg-Scopes ist, wird das Flash-Konfigurationsartefakt mit dem bestehenden Hetzner/Borg-Backup historisiert. Downloadbare Plugin-Paketarchive unter `/boot/config/plugins/*/` werden aus dem Artefakt ausgeschlossen; Restore-relevante Konfiguration bleibt enthalten.
### 2026-05-25 - Monitoring-Zielstack finalisiert und Uptime Kuma entfernt ### 2026-05-25 - Monitoring-Zielstack finalisiert und Uptime Kuma entfernt
- `monitoring` und `glance` wurden auf Commit `b6bbca4` deployed; Komodo zeigt fuer beide `latest_hash` = `deployed_hash` = `b6bbca4` ohne `remote_errors`. Die zehn `monitoring-*` Container laufen, `monitoring.kaleschke.info` und `glance.kaleschke.info` leiten anonym zu Authelia, Prometheus ist ready und Loki `/ready` liefert `ready`. - `monitoring` und `glance` wurden auf Commit `b6bbca4` deployed; Komodo zeigt fuer beide `latest_hash` = `deployed_hash` = `b6bbca4` ohne `remote_errors`. Die zehn `monitoring-*` Container laufen, `monitoring.kaleschke.info` und `glance.kaleschke.info` leiten anonym zu Authelia, Prometheus ist ready und Loki `/ready` liefert `ready`.
+2
View File
@@ -26,6 +26,7 @@ Sie ist die fachliche Ergaenzung zu `docs/DISASTER_RECOVERY.md`.
| Dienst | Fuehrende Quelle | Datei-Restore | Dump / DB | Secrets / ENV | Abhaengigkeiten | Smoke-Test | | Dienst | Fuehrende Quelle | Datei-Restore | Dump / DB | Secrets / ENV | Abhaengigkeiten | Smoke-Test |
|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|
| Unraid OS Flash | Borg-Artefakt + optional Unraid Connect | `/boot/config` aus `unraid-flash-config.tar.gz` | `unraid-flash-config.tar.gz`, `.sha256`, Manifest | enthaelt sensible Host-Konfiguration, wie Secret-Material behandeln | Unraid USB Flash Creator / neuer Boot-Stick | Unraid bootet, Array-Zuordnung und Shares sind sichtbar |
| Traefik | Share / Borg | `/mnt/user/appdata/traefik`, besonders `dynamic/`, `letsencrypt`, `secrets` | keine eigene DB | `cloudflare_dns_api_token` | `frontend_net`, `backend_net` | `https://traefik.kaleschke.info` erreichbar, Dashboard ueber Authelia | | Traefik | Share / Borg | `/mnt/user/appdata/traefik`, besonders `dynamic/`, `letsencrypt`, `secrets` | keine eigene DB | `cloudflare_dns_api_token` | `frontend_net`, `backend_net` | `https://traefik.kaleschke.info` erreichbar, Dashboard ueber Authelia |
| AdGuard Home | Share / Borg | `/mnt/user/appdata/adguard/conf` | keine | keine zusaetzlichen Repo-Secrets dokumentiert | `dns_net`, `frontend_net` | DNS-Aufloesung funktioniert | | AdGuard Home | Share / Borg | `/mnt/user/appdata/adguard/conf` | keine | keine zusaetzlichen Repo-Secrets dokumentiert | `dns_net`, `frontend_net` | DNS-Aufloesung funktioniert |
| Tailscale | Share / Borg | `/mnt/user/appdata/tailscale` | keine | Tailscale-State im Pfad | Host-Netz | Tailscale verbunden | | Tailscale | Share / Borg | `/mnt/user/appdata/tailscale` | keine | Tailscale-State im Pfad | Host-Netz | Tailscale verbunden |
@@ -90,6 +91,7 @@ Aktuell relevante Dump-Artefakte unter `/mnt/user/backups/borg/dumps/latest`:
- `filebrowser.bolt.dump` - `filebrowser.bolt.dump`
- `borg-ui.sqlite` - `borg-ui.sqlite`
- `grafana.sqlite` - `grafana.sqlite`
- `unraid-flash-config.tar.gz` plus `unraid-flash-config.tar.gz.sha256` und Manifest
- Monitoring-Stack: keine verpflichtenden Dump-Artefakte; Prometheus/Loki/Grafana named volumes sind Diagnose-/Dashboard-Zustand, keine primaere Restore-Quelle. - Monitoring-Stack: keine verpflichtenden Dump-Artefakte; Prometheus/Loki/Grafana named volumes sind Diagnose-/Dashboard-Zustand, keine primaere Restore-Quelle.
- `komodo-mongo.archive.gz` (noch gesondert verifizieren) - `komodo-mongo.archive.gz` (noch gesondert verifizieren)
+1
View File
@@ -46,6 +46,7 @@ Dieses Dokument listet sensible Daten, deren Ablageorte und die vorgesehene Einb
| nextcloud-postgres | DB Password | `/mnt/user/appdata/secrets/nextcloud_postgres_password.txt` -> `POSTGRES_PASSWORD_FILE` | neu | | nextcloud-postgres | DB Password | `/mnt/user/appdata/secrets/nextcloud_postgres_password.txt` -> `POSTGRES_PASSWORD_FILE` | neu |
| Borg UI / Borg | Admin-Login, `SECRET_KEY`, SSH-Keys, Repo-Credentials | persistent unter `/mnt/user/appdata/borg-ui/data/` | aktiv | | Borg UI / Borg | Admin-Login, `SECRET_KEY`, SSH-Keys, Repo-Credentials | persistent unter `/mnt/user/appdata/borg-ui/data/` | aktiv |
| Borg Repo | Borg-Passphrase fuer Restore-Tests und Notfallzugriff | `/mnt/user/appdata/secrets/borg_repo_passphrase.txt` -> Host-Secret-Datei, nicht im Repo | aktiv | | Borg Repo | Borg-Passphrase fuer Restore-Tests und Notfallzugriff | `/mnt/user/appdata/secrets/borg_repo_passphrase.txt` -> Host-Secret-Datei, nicht im Repo | aktiv |
| Unraid Flash Backup | Boot-/Array-/Share-/Plugin-Konfiguration, ggf. Hashes/Keys/Templates | `/mnt/user/backups/borg/dumps/latest/unraid-flash-config.tar.gz`, via Borg/Hetzner gesichert | aktiv; wie Secret-Material behandeln |
| Hermes Agent | Provider-Keys, Bot-Tokens, API-Server-Key | `/mnt/user/appdata/hermes-agent/data/.env` | VM-seitig offen | | Hermes Agent | Provider-Keys, Bot-Tokens, API-Server-Key | `/mnt/user/appdata/hermes-agent/data/.env` | VM-seitig offen |
| Hermes Agent | SSH-Runner Private Key | `/mnt/user/appdata/secrets/hermes_runner_id_ed25519` -> `/root/.ssh/id_ed25519` | VM-seitig offen | | Hermes Agent | SSH-Runner Private Key | `/mnt/user/appdata/secrets/hermes_runner_id_ed25519` -> `/root/.ssh/id_ed25519` | VM-seitig offen |
| Grafana | Admin Password | `/mnt/user/appdata/secrets/grafana_admin_password.txt` -> `GF_SECURITY_ADMIN_PASSWORD__FILE` | aktiv | | Grafana | Admin Password | `/mnt/user/appdata/secrets/grafana_admin_password.txt` -> `GF_SECURITY_ADMIN_PASSWORD__FILE` | aktiv |
+5 -1
View File
@@ -11,16 +11,18 @@ Use Borg as the single backup system for:
- critical file-backed application data - critical file-backed application data
- secrets, keys, and reverse-proxy state - secrets, keys, and reverse-proxy state
- database dumps generated before each Borg backup - database dumps generated before each Borg backup
- Unraid flash configuration artifacts generated before each Borg backup
Do not back up raw live database storage directories as the primary recovery artifact. Do not back up raw live database storage directories as the primary recovery artifact.
## Strategy ## Strategy
1. A pre-backup dump script runs on the host and writes fresh dumps to `/mnt/user/backups/borg/dumps/latest`. 1. A pre-backup dump script runs on the host and writes fresh dumps plus `unraid-flash-config.tar.gz` to `/mnt/user/backups/borg/dumps/latest`.
2. Borg backs up `/local/borg-dumps` plus the critical mounted paths below. 2. Borg backs up `/local/borg-dumps` plus the critical mounted paths below.
3. Borg retention handles history; the dump directory itself keeps only the latest artifacts. 3. Borg retention handles history; the dump directory itself keeps only the latest artifacts.
The inclusion of `/local/secrets` is intentional: Borg is expected to cover disaster recovery for selected secret material as part of the current homelab restore strategy. The inclusion of `/local/secrets` is intentional: Borg is expected to cover disaster recovery for selected secret material as part of the current homelab restore strategy.
The Unraid flash configuration archive is intentional as well and must be treated as secret backup material.
## Service Inventory ## Service Inventory
@@ -41,6 +43,7 @@ The inclusion of `/local/secrets` is intentional: Borg is expected to cover disa
| Borg UI | SQLite dump + self-backup | `/local/borg-dumps`, `/local/appdata/borg-ui/data` | | Borg UI | SQLite dump + self-backup | `/local/borg-dumps`, `/local/appdata/borg-ui/data` |
| Komodo | config + Mongo dump | `/local/borg-dumps`, `/local/appdata/komodo/periphery`, `/local/appdata/komodo/core` | | Komodo | config + Mongo dump | `/local/borg-dumps`, `/local/appdata/komodo/periphery`, `/local/appdata/komodo/core` |
| GitOps host automation | repo clone + Komodo workspaces + host-check state | `/local/services/homelab-infra`, `/local/services/stacks`, `/local/services/posture-check` | | GitOps host automation | repo clone + Komodo workspaces + host-check state | `/local/services/homelab-infra`, `/local/services/stacks`, `/local/services/posture-check` |
| Unraid OS flash | generated config archive | `/local/borg-dumps/unraid-flash-config.tar.gz` plus checksum and manifest |
| Nextcloud | DB dump + file data | `/local/borg-dumps`, `/local/appdata/nextcloud/html`, `/local/nextcloud/data` | | Nextcloud | DB dump + file data | `/local/borg-dumps`, `/local/appdata/nextcloud/html`, `/local/nextcloud/data` |
| Grafana | SQLite dump + file data | `/local/borg-dumps`, `/local/appdata/grafana` | | Grafana | SQLite dump + file data | `/local/borg-dumps`, `/local/appdata/grafana` |
| Filebrowser | file-backed state dump + file data | `/local/borg-dumps`, `/local/appdata/filebrowser` | | Filebrowser | file-backed state dump + file data | `/local/borg-dumps`, `/local/appdata/filebrowser` |
@@ -83,6 +86,7 @@ The live Unraid User Scripts execute repo scripts from `/mnt/user/services/homel
- Komodo MongoDB - Komodo MongoDB
- SQLite: `gitea`, `vaultwarden`, `speedtest-tracker`, `borg-ui`, `grafana` - SQLite: `gitea`, `vaultwarden`, `speedtest-tracker`, `borg-ui`, `grafana`
- File-backed state: `filebrowser.bolt.dump` - File-backed state: `filebrowser.bolt.dump`
- Unraid flash config: `unraid-flash-config.tar.gz` plus `unraid-flash-config.tar.gz.sha256`
## Explicitly Not Backed Up as Raw Live DB Files ## Explicitly Not Backed Up as Raw Live DB Files
+4
View File
@@ -14,6 +14,10 @@ Fresh dump artifacts are written to:
Borg UI should include `/local/borg-dumps` as a backup source. Borg UI should include `/local/borg-dumps` as a backup source.
The dump set also includes `unraid-flash-config.tar.gz`, a host-generated
archive of `/boot/config` plus checksum and manifest. Treat this archive as
secret backup material.
## Notes ## Notes
- The script is written for host execution where `docker` is available. - The script is written for host execution where `docker` is available.
+5 -4
View File
@@ -17,7 +17,7 @@ It should **not** be implemented as a Borg UI inline hook in the current design.
`pre-borg.sh` currently chains the host-side checks: `pre-borg.sh` currently chains the host-side checks:
- `services/posture-check/posture-check.sh` - `services/posture-check/posture-check.sh`
- `ops/borg-ui/scripts/pre-backup-dumps.sh` - `ops/borg-ui/scripts/pre-backup-dumps.sh` including the Unraid flash config archive
- `ops/restore-tests/check-restore-freshness.sh` - `ops/restore-tests/check-restore-freshness.sh`
The dump step assumes: The dump step assumes:
@@ -56,9 +56,10 @@ The intended sequence is:
1. Host wrapper checks posture. 1. Host wrapper checks posture.
2. Host script refreshes `latest` dump artifacts. 2. Host script refreshes `latest` dump artifacts.
3. Freshness check verifies expected dumps. 3. Host script writes `unraid-flash-config.tar.gz` plus checksum and manifest into the same dump set.
4. Borg UI backs up `/local/borg-dumps` together with the rest of `critical_infra`. 4. Freshness check verifies expected dumps and the flash config archive.
5. Borg history preserves dump history, so the host only needs to keep the most recent dump set. 5. Borg UI backs up `/local/borg-dumps` together with the rest of `critical_infra`.
6. Borg history preserves dump history, so the host only needs to keep the most recent dump set.
## Current dump target ## Current dump target
+56
View File
@@ -155,6 +155,56 @@ dump_file_copy() {
atomic_write "$output" "$tmp" atomic_write "$output" "$tmp"
} }
backup_unraid_flash_config() {
output="$LATEST_DIR/unraid-flash-config.tar.gz"
checksum="$LATEST_DIR/unraid-flash-config.tar.gz.sha256"
manifest="$LATEST_DIR/unraid-flash-config.manifest.txt"
tmp="$TMP_DIR/unraid-flash-config.tar.gz.tmp"
tmp_checksum="$TMP_DIR/unraid-flash-config.tar.gz.sha256.tmp"
tmp_manifest="$TMP_DIR/unraid-flash-config.manifest.txt.tmp"
if [ ! -d /boot/config ]; then
warn "Skipping Unraid flash config backup because /boot/config is missing"
return 1
fi
log "Backing up Unraid flash configuration from /boot/config"
rm -f "$tmp" "$tmp_checksum" "$tmp_manifest"
tar -C /boot \
--exclude='config/plugins/*/*.txz' \
--exclude='config/plugins/*/*.tgz' \
--exclude='config/plugins/*/*.tar' \
--exclude='config/plugins/*/*.tar.*' \
--exclude='config/plugins/*/*.zip' \
--exclude='config/plugins/*/*.md5' \
-czf "$tmp" config
chmod 600 "$tmp"
atomic_write "$output" "$tmp"
(
cd "$LATEST_DIR"
sha256sum "$(basename "$output")"
) > "$tmp_checksum"
chmod 600 "$tmp_checksum"
atomic_write "$checksum" "$tmp_checksum"
{
printf 'created_utc=%s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
printf 'host=%s\n' "$(hostname)"
if [ -f /etc/unraid-version ]; then
sed 's/^/unraid_/' /etc/unraid-version
fi
printf 'source=/boot/config\n'
printf 'archive=%s\n' "$(basename "$output")"
printf 'checksum=%s\n' "$(basename "$checksum")"
printf 'note=%s\n' 'Contains Unraid configuration and must be treated as secret backup material.'
printf 'excluded=%s\n' 'downloadable plugin package archives under /boot/config/plugins/*/'
} > "$tmp_manifest"
chmod 600 "$tmp_manifest"
atomic_write "$manifest" "$tmp_manifest"
}
dump_optional_pg_db() { dump_optional_pg_db() {
container="$1" container="$1"
password="$2" password="$2"
@@ -219,6 +269,8 @@ dump_mongo_container() {
main() { main() {
need_cmd docker need_cmd docker
need_cmd sqlite3 need_cmd sqlite3
need_cmd tar
need_cmd sha256sum
ensure_dirs ensure_dirs
# Shared PostgreSQL 17 # Shared PostgreSQL 17
@@ -272,6 +324,10 @@ main() {
# MongoDB # MongoDB
dump_mongo_container "komodo-mongo" "$LATEST_DIR/komodo-mongo.archive.gz" dump_mongo_container "komodo-mongo" "$LATEST_DIR/komodo-mongo.archive.gz"
# Unraid USB flash configuration. This is generated into the existing dump
# set so Borg carries it off-site together with the database artifacts.
backup_unraid_flash_config
log "Finished refreshing dump set in $LATEST_DIR" log "Finished refreshing dump set in $LATEST_DIR"
} }
@@ -14,7 +14,8 @@ $checks = @(
@{ Name = "gitea.sqlite.dump"; Path = Join-Path $DumpRoot "gitea.sqlite.dump" }, @{ Name = "gitea.sqlite.dump"; Path = Join-Path $DumpRoot "gitea.sqlite.dump" },
@{ Name = "vaultwarden.sqlite.dump"; Path = Join-Path $DumpRoot "vaultwarden.sqlite.dump" }, @{ Name = "vaultwarden.sqlite.dump"; Path = Join-Path $DumpRoot "vaultwarden.sqlite.dump" },
@{ Name = "speedtest-tracker.sqlite.dump"; Path = Join-Path $DumpRoot "speedtest-tracker.sqlite.dump" }, @{ Name = "speedtest-tracker.sqlite.dump"; Path = Join-Path $DumpRoot "speedtest-tracker.sqlite.dump" },
@{ Name = "filebrowser.bolt.dump"; Path = Join-Path $DumpRoot "filebrowser.bolt.dump" } @{ Name = "filebrowser.bolt.dump"; Path = Join-Path $DumpRoot "filebrowser.bolt.dump" },
@{ Name = "unraid-flash-config.tar.gz"; Path = Join-Path $DumpRoot "unraid-flash-config.tar.gz" }
) )
$reportChecks = @( $reportChecks = @(
+2 -1
View File
@@ -34,7 +34,8 @@ for dump in \
gitea.sqlite.dump \ gitea.sqlite.dump \
vaultwarden.sqlite.dump \ vaultwarden.sqlite.dump \
speedtest-tracker.sqlite.dump \ speedtest-tracker.sqlite.dump \
filebrowser.bolt.dump; do filebrowser.bolt.dump \
unraid-flash-config.tar.gz; do
path="$DUMP_ROOT/$dump" path="$DUMP_ROOT/$dump"
if [ ! -f "$path" ]; then if [ ! -f "$path" ]; then
critical+=("DUMP_MISSING $dump") critical+=("DUMP_MISSING $dump")